waterfall data layer

collection.js

define([
  'underscore',
  'knockout'
], function(_, ko) {

  function Collection() {
  }

  Collection.prototype.init = function() {
    this._instances = ko.observableArray();
  };

  // to be overridden:
  // - Collection.prototype.url
  // - Collection.prototype.create
  // - Collection.prototype.update

  Collection.prototype._dedupe = function(item) {
    var existing = this.byId(item.id);
    if (!existing) {
      existing = this.create(item);
      this._instances.push(existing);
    } else {
      this.update(existing, item);
    }
    return existing;
  };

  Collection.prototype.byId = function(id) {
    return _.find(this._instances(), function(instance) {
      return instance.id() === id;
    });
  };

  Collection.prototype.fetch = function(params) {
    // the specific thing we're using for the ajax request should maybe be
    // abstracted
    return $.ajax(_.extend({
        url: this.url
      }, params))
      .then(function(result) {
        result.forEach(this._dedupe.bind(this));
        return result;
      });
  };

  Collection.prototype.fetchOne = function(id, params) {
    return $.ajax(_.extend({
        url: this._url + '/' + id
      }, params))
      .then(function(item) {
        return this._dedupe(item);
      });
  };

  // also need to figure out saving and deleting. two options:
  // - add them as methods to Collection.prototype (uglier)
  // - add them as methods to the items in resource._instances, but then we
  //   need a way of updating that list when an item is saved or deleted
  return Collection;
});

metadata-collection.js

define([
  'ko',
  'waterfall',
  'collection'
], function(ko, waterfall, Collection) {
  var instance;

  function MetadataCollection() {
    this.init();
  }
  
  MetadataCollection.prototype = new Collection();

  MetadataCollection.prototype.url = waterfall.metadata.url;

  // this is where things get weird. we probably want to return an instance of
  // something w/ a 'save' and 'delete' method, but those instances need some
  // way of notifying the Collection instance when they're deleted or saved for
  // the first time so the Collection can update its _instances OA accordingly
  //
  // so basically like a Record class
  MetadataCollection.prototype.create = function(item) {
    return {
      id: ko.observable(item.id),
      name: ko.observable(item.name),
      multiValue: ko.observable(item.multiValue),
      format: ko.observable(item.format)
    };
  };

  MetadataCollection.prototype.update = function(existing, item) {
    existing.id(item.id);
    existing.name(item.name);
    existing.multiValue(item.multiValue);
    existing.format(item.format);
  };

  // use an accessor function, since we'll usually just want a single shared
  // app-wide Collection of MetadataCollection, but we want to be able to reset
  // that instance for stuff like unit testing
  //
  // the "singleton pattern", or "object manager factory get instance function"
  function accessor() {
    return instance? instance : instance = new MetadataCollection();
  }

  accessor.MetadataCollection = MetadataCollection;

  accessor.reset = function() {
    instance = null;
  };

  return accessor;
});

a few things about these COA's:

• yes, they need to be more complicated than simply a computed that returns an array because we want to bind them to UI elements, and be able to insert items into the underlying array w/o re-rendering the entire list. in order for this to happen, knockout needs to be binding to an actual OA
• the COA's i linked to can be thought of as _.map in that they just take some translation from the elements of the source array to the elements of the computed array, but they maintain the order. sorting is a different case, since the items remain the same, but the order changes. so if we want to support both, we'll have to re-think that implementation

Just took a stab at a generic COA implementation based on your filters and this discussion.

It accepts any compute function, so it should work for sorting as well as mapping translations. There's an example in the comments there for using it to sort the metadata collection alphabetically by name.

It uses a helper function mergeObservableArrays that takes the diff of two observable arrays and tries to merge them with minimal removes and inserts. We can work to optimize this algorithm over time.

The code probably doesn't work because, like any good developer, I didn't actually test it out. 😄 A working implementation shouldn't be far off though.

Why would we want to rely on the server for what we could easily accomplish with...

great question. 2 scenarios:

1. we have a view w/ a query of the 1st 20 items, then the user creates a new item and saves it. should this query include that new item? if so, which should it exclude (presumably the view is a table of 20 rows)? and how does this differ from one resource to the next? i.e. the 1st 20 of one resource may be determined by sorting the name field, while another is determined by sorting the id

2. the user vists /metadata/page/2, and then clicks a /metadata link. the state of the metadataCollection._items OA will be one of 2 things:

   1. a simple array of indices 0-19
   2. a sparse array w/ nothing in 0-19, and the metadata models in indices 20-39

   if it's the first case, then how does the Query instance for the /metadata page know that the stuff in metadataCollection._items corresponds to the second page?

   if it's the second case, then we've got to do a bunch of bookkeeping when we insert/remove models or change the sorting key.

i'm not saying these are impossible to solve, but they're non-trivial, hard to hammer all of the bugs out of, and hard to design in a way that's robust to system changes. so the question is what we get by trying to get this right, versus just making that API call every time we want a Query. i don't think the small perf benefits outweigh the cost of implementation, especially if this is something we can just implement later (i.e. our current design doesn't preclude this).

so a couple questions to get us on the same page:

• are we seeing the same cost of implementation? like when i think of keeping Query's in sync w/ server state i see that as a big task. do you?
• are we seeing the same benefit? the only benefit i see is the user will be able to go to a number of pages w/o waiting for the api call, which doesn't seem like much. am i missing other benefits tho?

i'll add comments to the COA gist.

Really productive talk today @aaronj1335! 👍 Some notes:

• Queries aren't computed observable arrays, but each one has an underlying (non-computed) observable array
• Collections and queries each have their own observable array, but they share model instances
• When a query calls the server, it gets updated with the appropriate model instances. The collection corresponding to that type of model also gets updated. If those model instances aren't in the collection yet, they're added. If they are in the collection, then they're updated with the new data from the server.
• Collections aren't bootstrapped at the initial page load. They just grow organically throughout the session as queries are made to the server.
• Computed array observables aren't needed for the initial implementation of this data layer
• Queries can be created, refreshed and deleted, but they can't be changed to different types of queries (queries with different parameters)

This stuff is feeling much more clear to me now. Planning to sketch out some code and/or detailed examples soon.

Now for the fun part: what should we name this little library we're developing? 😄 Knockout Data? Boxing Gloves? (Because you wouldn't want to knock someone out without boxing gloves!)

Started a branch for this called "data-layer" in WaterfallEngineering/frontend.