gistfile1.md

DataTable design overview

These are my early thoughts on how to structure DataTable in 3.5.0. Feedback is welcome in the comments.

Instantiable classes out of the box

new Y.DataTable({...});
new Y.DataTable.Base({...});

Class generation

Y.DataTable.Base = Y.Base.create('datatable', Y.Widget, [Y.DataTable.Core]);
Y.DataTable = Y.mix(
    Y.Base.create('datatable', Y.DataTable.Base, []),
    Y.DataTable, true);

Class extensions CAN (if non-invasive by default)

Y.Base.mix(Y.DataTable, [ Y.DataTable.Sortable ]);

Primary configuration

new Y.DataTable({
    data: [{ ... }, { ... }, ... ]
    data: modelList
    data: {
        source: url, function, xml
        type: 'json'
        schema: { ... }        
    }

    columns: (optional) [ key, key, { key: key, config: stuff, ... }, ... ]

    recordType: (optional) ModelClass

    headerView: (optional) ViewClass
    bodyView: (optional) ViewClass
    footerView: (optional) ViewClass

    summary: (optional) 'string'
    caption: (optional) 'string'
});

Instance structure

instance
    .data = modelListInstance

    .head = viewInstance
    .body = viewInstance
    .foot = viewInstance

Component classes

• Y.DataTable.Core

  Everything added to Y.Widget to make DataTable.Base

• Y.DataTable.HeaderView

• Y.DataTable.BodyView

• Y.DataTable.FooterView

  Used by DataTable(.Base) to render the table contents.

  Referenced via configuration, not composition.

• Y.DataTable.Source

  Optional class extension.

  Adds support data config to generate ML with DataSource load.

• Y.DataTable.Scrollable

  Optional class extension.

  Adds support for scrolling table. May be used to create a third instantiable class.

• Y.DataTable.ScrollingBodyView

  Used in place of DataTable.BodyView to render the scrolling table. May not be necessary?

• Y.DataTable.Sortable

  Adds support for sorting headers and sortable config.

Render lifecycle

instance.render(...)

1. build <table> (markup only or off DOM node?)
2. instantiate header, body, footer views passing
   1. the DT instance
   2. the created table?
   3. the data ModelList?
3. call each view's render()

Logic for reacting to user input is moved to the View classes

Concern: string based rendering would have viewX.render() return a string or populate a property, but it has requirements of the DT renderer

Load lifecycle

instance.load(...)

Pass through to this.data.load(...). Let the ModelList be responsible for data interaction.

Default Model generation

• No ModelList is provided in data configuration AND
• No recordType is provided in configuration

1. Use the first item in the data array to build ATTRS and Y.Base.create(...)
2. If data is empty, use columns?

I thought I made it fairly clear in the gist:

The fact that they derive from Y.View is clear.

What I was trying to ask was will they extend, as in, add to, the API, and how:

c) Are the default Views going to extend the base View implementation? Either way, it may be worth documenting that in the design docs so there's some idea of the kind of API they may have.

Since the Y.View API is generic. I'm trying to get an idea of whether you're breaking them out just for the base View functionality [ template/render/container ] or is there additional model-bound API to them.

Which leads into the other question...

Not sure what you mean.

Views can (optionally) have references to Model/ModelLists.

Related to the above question about the default Head/FootView API, I was asking if they would have a reference to the Model/ModelList driving the core DT Data - and how the View API would interact with that Model/ModelList reference.

I'm trying to get a handle on what kind of API the HeadView, FootView will have, and if/how they bind to a Model. Or whether they are just encapsulated "renderer" implementations without any interaction with the Model/ModelList driving the Data.

The answer to both questions probably boils down to what the Head/FootView API will look like, and how (if at all) it responds to changes in the Model.

@sdesai

will they extend, as in, add to, the API, and how

Aha, Gotcha. I'm not planning on relying on any additional API at this point. When I get into the implementation details, in particular when I start working on feature modules, some DT related render/bind related methods might emerge as being helpful or even a soft requirement for custom Views targeting DT consumption. I'll report back when I get into that (should be Mon/Tue).

[ template/render/container ] or is there additional model-bound API to them

I expect the Views to be able to respond to data changes and update the portions of the table they are responsible for, but that seems like expected View behavior. I am anticipating a need for some sort of uiUpdate custom event (or suite of common events) fired from each of the views when they apply DOM updates in response to user interaction or data modification. But again, I'm not there yet, so I'll report back.

if/how they bind to a Model. Or whether they are just encapsulated "renderer" implementations

At this point, I do plan on having the View instances bound to the data ML, and possibly (though loosely if necessary) to the DT itself.

@mosen I agree with this, and think it is a good idea:

With regards to placing yuid's on table elements: I ran with clientIds as the TR id, and kept the headers="" attribute that was added to each cell to indicate which column it was part of. Using the clientId on the row makes it pretty easy to translate table events back into an item from your model list, if you want some kind of change to occur on the model.

@lsmith

I expect the Views to be able to respond to data changes and update the portions of the table they are responsible for, but that seems like expected View behavior. I am anticipating a need for some sort of uiUpdate custom event (or suite of common events) fired from each of the views when they apply DOM updates in response to user interaction or data modification. But again, I'm not there yet, so I'll report back.

I think if you can, you should try to avoid this two-way syncing. I think the custom View events are the right way to go, but I think they should merely inform the DT about what changes to make the to Models, and have the DT carry out those changes (one source of truth).

It seems like you probably want the DT to be a bubble target for the View events, so that the View-related custom events bubble back to the DT, and the DT can subscribe to them in a loose manner.

It also seems like you'll want to pass/set the ModelList to these Views so they can respond to changes as @sdesai was saying. I can imagine a FooterView that displays the total number of records my calling this.get('modelList').size() and rendering the result.

@ericf

I think if you can, you should try to avoid this two-way syncing.

Agreed, thanks for clearing my head up. The DT instance will serve in a more formal Controller role. Views will subscribe to Model events to propagate changes to the UI, and to UI events to report user input to the DT/Controller, which then updates the Models. It's like an MVC pattern, or something!

DT to be a bubble target for the View events

Yes

pass/set the ModelList to these Views so they can respond to changes

That was the plan, yes.

@mosen Thanks for the feedback!

Extensions that affect head and body views such as Sortable/Plugins weren't aware of each other

Definitely I'll make sure the two are compatible. As to the larger issue of plugins ont being aware of one another, I'm hoping the uiChange event (or suite of events) will help bridge the communication gap, because, you're right, they should not be aware of one another by design. I'm hoping there will be few cases where this manifests, but I'm hoping that the new structure will help isolate the conflict areas. The proof in the pudding will come when I start working on features (PR2 and 3).

there has to be a standard way to access every part of the table from the base

That's part of the reason for having the head, body, foot properties, because the APIs that appropriate for a specific row visualization can be sequestered onto the View class. If there's a reasonable, consistent way to expose the tbody or tbodies (I don't see a problem with the thead or tfoot) that will support the wide variety of row visualizations, then I'm all for it. I'm open to suggestions.

Using the clientId on the row

I agree with @ericf that this sounds like a good idea. I worry this will conflict with the flyweight ML, though, since there wouldn't be a permanent Model instance for each row. I don't want to reinvent any wheels.

updates of single models or single fields is much much better (obviously) than RecordSet causing the table to re-render. Iterating through the modelList and creating the nodes did cause a bit of a performance problem, probably most noticeable due to multiple calls of .get(field). flattening the attributes would help (as would a flyweight modellist).

Yeah, it's high on my performance related priority list that record/Model changes will affect isolated UI changes. That's run time behavior. The hit coming from Attribute during render time is definitely an issue. Having the record data in Attributes gives us a single place to define extra behavior like setters (aka parser) and validators and getters (formatters? --seems like View functionality), but is costly for what I would suspect is the majority use case of having a simple data map like a POJO. I have some incomplete ideas on this that I'll enumerate in a separate comment or gist.

it was the individual calls to model and in some cases formatters

The current structure has each cell going through several functions in the render process. Actually, all cells go through a formatter, just most through the default one. I'm pretty sure we can avoid a lot of these function calls.

At the moment I settled on the formatter returning a string only, but some people might need to set TD attributes somehow. I guess if you have control over the entire template via the BodyView this doesn't become such a problem.

Yep, that's the idea. If time allows, I'd like to create a default BodyView that uses exclusively string concatenation, but provide another that supports cell Node creation. The latter is more likely to be in 3.5.0 for backward compatibility.

I would propose maybe two kinds of formatters? (this is assuming the way it builds rows even resembles the previous process at all). One to formulate the value of a cell (because sometimes they don't bear a 1:1 relation to the model attribute), and one to format the visual representation of the cell data.

I don't follow. Can you give an example of when there would need to exist a value state for a cell that was neither the attribute value or the rendered cell data?

One design that could alleviate the formatting and node construction issues would be to have a 1:1 relationship between a single model and a view template. I guess then you could throw the smarts into the view, and also cater for a multi-row representation of a single model.

This was part of the reason to have the bodyView break out. One View implementation can use row templates + Y.Lang.sub(), another could use Node creation, and another could have sub-Views for things like expandable rows. I sure hope it works out!

cue lsmith telling me to go away and make it a gallery module :)

Go make a gallery module. What are we talking about? Eh, doesn't matter. Go make a gallery module!

Similar to the bodyView for custom implementations, make a custom headView.

@stlsmiths
#1 - Pagination is in the plan for 3.5.0. Data syncing with the server will be delegated to the ModelList.
#2 - The specific implementations haven't been worked out, but I plan on taking full advantage of the work @rgrove and @ericf have been doing in the App architecture.
#3 - Per my reply to @mosen, I'm definitely planning to have individual model updates only touch that part of the UI. No full table rerenders for this DT!
#4 - I think I covered my initial thoughts on this in the reply to @mosen, but yeah, official grouping support is targeted for 3.7.0

Keep the feedback coming!

@mundizzle

100% width has been a thorn in the side of DT for a long time. It's mostly (exclusively?) been an issue with scrollable tables, and I have that specific task targeted for 3.7.0.

I don't have any plans for split pane functionality, unless this is another way of saying freeze columns, like the Excel functionality. That's another one that's been asked for a lot (it never existed for YUI 2, either). I have it loosely targeted for 3.8.0, but the big issue has been accessibility. We'll see when we get closer. I'm sure a lot will have changed by then. In the mean time, all suggestions and feedback are welcome.

@lsmith sorry I was a bit vague with the dual formatters idea.
It's something I ran across, but now I think that it's out of the scope of your development (wrt calculation in footer elements).

Hi,

Awesome stuff, fantastic to see YUI3 DT seeing more activity =).

Apologies if this is slightly off-topic, however I've found that the YUI3 Datatable examples are quite spartan:

http://yuilibrary.com/yui/docs/datatable/

In contrast, the YUI2 examples:

http://developer.yahoo.com/yui/datatable/

contain examples of how to do pagination, client-side filtering by buttons, auto-complete searching, in-line cell editing etc.

I'm guessing most of this is still achievable with YUI3 DT, right?

Is there any chance that the YUI3 examples might be updated sometime to reflect how to do the above with DT now? It'd be great to have some code examples to follow along to.

Cheers,
Victor

@victorhooi YUI 3 DT isn't as feature rich as YUI 2's DT yet. The first milestone is getting a solid foundation for the core functionality, then start working on the features. That's also why there are fewer examples for YUI 3 DT.

3.5.0 DT work will focus on revamping the architecture (as you can tell from this gist), but I'm also planning to add Pagination. Editing, highlighting, and selection are targeted for 3.6.0, then expandable rows and other features in subsequent releases.

WRT examples, though, as of 3.4.0, it's possible for anyone to create examples. The example files live in the src/{component}/docs/ directory in the yui3 project repo on github (https://github.com/yui/yui3) and are built using HandleBars templates compiled with Selleck (https://github.com/rgrove/selleck). I'll be adding examples over time, but pull requests are welcome!

@ericf, @mosen - The row id == clientId came out because I worry that the clientId won't be unique across DataTable instances or YUI instances.

@sdesai - so far, I've seen a need for the body and header views to have a getClassName method (that I cribbed from Widget) and the body view has a getCell and getRow method which the Core implementation relays the arguments to. Since it is up to the view to decide how to create the rows and cells, it should be up to the view to decide how to fetch them.

The getClassName being on the views is mainly to keep the views decoupled from DataTable. I pass the _cssPrefix to the views as a cssPrefix config property. The views' initializer then takes the config and, if set, sets the instance's _cssPrefix, shadowing the default _cssPrefix from the view's prototype. This allows the view instances to use their own getClassName to generate classes appropriate for the DT or for another consumer, falling back to classes built from the prototype _cssPrefix. I can either keep you updated here on in the pull request (yui/yui3#63).

@lsmith: Model clientIds are guaranteed to be unique by default for all model instances on the same page (even across sandboxes), so that should work fine. It's specifically intended for cases like this, where you want to create a DOM id for something related to a model. The only catch would be if someone overrides generateClientId() on a model, has it generate a non-unique id, and then inserts that model into a DataTable. But that's a pretty clear case of taking careful aim and then shooting oneself in the foot, so I wouldn't worry about it.

Regarding the need for getClassName on views: hmm. I actually closed a ticket for this as wontfix earlier today because I didn't think it was worth giving View a dependency on ClassNameManager, but if you think it's valuable, I can consider it.

@rgrove Cool, I'll put clientId back in then.

Maybe I'm using View wrong, or maybe View isn't the right base class. In its role as a renderer for a portion of a Widget, it's not really a one-off, but a utility for library code, so naked class names (without the leading yui3-) didn't seem appropriate. That's why I went with getClassName. At this point, I don't think I'm using much of View's API. And things like initializer calling create() commits me to working with Nodes perhaps earlier than I wanted to. I can override that logic, but at what point am I inheriting from the class just to inherit from the class? Maybe View is better as a subclass for final implementation logic rather than as a middle layer class for components. Wasn't that its initial purpose?

The value of View for DataTable is in providing an API contract. By default, DataTable's views render things a certain way. By subclassing one of DataTable's views and customizing it, a user can easily swap out the default behavior for a view that uses Handlebars for rendering, or that defines custom interactions, or any number of other things. A user could even use one set of views to render DataTables on the client, and another set of Views (but the same underlying DataTable code) to render them on the server. Or one set on desktop, another on mobile. And as users begin to write View extensions and plugins, DataTable's views will benefit from them.

The fact that DataTable doesn't use much of View's API doesn't seem relevant (especially given that View doesn't have much of an API); what's relevant is that by using Views, DataTable is opening up a world of extensibility possibilities that would otherwise be more difficult to achieve.

@rgrove I'm not arguing to subsume the content rendering into the DT widget code. I'm happy with the .head, .body, and .foot breakout. I'm suggesting that a superclass whose API provides your subclass no value is a questionable choice for a superclass. If I'm not using View's API, perhaps I should be subclassing Base directly.

The argument is theoretical because I am still using some of the API, and so far, the rest hasn't cost me anything except the Node creation requirement, which is fine on the client. Renderers for node.js would need to work around this. The whole thing is moot anyway, because I'm not requiring headerView, bodyView, and footerView to be View instances. I use duck typing. They could be simple objects with the required API for all DT cares.

The question was about if getClassName was appropriate to live on View or should it be implemented on subclasses that need it. I'm fine leaving the implementation in my subclasses, and if there's further call for it from the community, then View can adopt it and I'll remove my implementations. The only awkward part IMO is the passing of the _cssPrefix so my views can generate classes as if they were part of the Widget code directly. If that use case should be supported if getClassName is added to View is the more interesting question to me.

Again, the value of View for DataTable is less about what DataTable can get from it and more about what DataTable's users can get from it. Even so, I'm curious why you don't find View's event delegation valuable. Seems perfect for DataTable.

Regarding the Node creation requirement: there isn't one. View's create() method creates a Node by default, but it's intended to be easily overridden if you want it to do something else. Ditto the attachEvents() method. Also, create() isn't called by View's initializer() method, it's called because it's the setter for the container attribute. You can easily override the attribute config without having to override the entire initializer.

The event delegation will be useful to features, I suspect, but the core behavior of the views doesn't involve user events.

This actually raises an interesting issue: What is the cleanest way for a DT class extension, for sorting say, to add UI interaction hooks?

My initial thought was DT class extension adds method sort(...) to DT's proto, and plugs in a table sorting plugin to the headerView instance.

Since the class responsible for rendering the header is configurable, and I'm trying to keep as much knowledge of the DOM structure out of DataTable as possible, it's an unsafe assumption to modify the prototype of any one View class (presumably Y.DataTable.HeaderView). That's what led me to plugins. However, plugins don't modify their hosts, which means the sortable extension wouldn't add to the view's events config.

I'll be dealing with this challenge specifically later this week or early next week, so more on that then.

This actually raises an interesting issue: What is the cleanest way for a DT class extension, for sorting say, to add UI interaction hooks?

Custom events. I'm a big believer that the View should translate DOM events, which resulted from user interactions, into higher-level custom events; in this case custom events which make sense in the world of DataTable.

I also feel that your Views should add the DT Widget as a bubble target. Then, the sorting extension can add a listener for some documented custom event, the View can fire that event, which will kick-off the sorting logic.

var SomeSortExt, CustomHeader;

SomeSortExt = function () {};
SomeSortExt.prototype = {
    initializer: function () {
        this.on('*:toggleSort', this.sort);
    },

    sort: function () {
        // Do the sorting by calling the ML's `sort()` method or something...
    }
};

CustomHeader = Y.Base.create('customHeader', Y.View, [], {
    events: {
        'th': {
            click: 'toggleSort'
        }
    },

    render: function () {
        this.get('container').append('<button class="sort">Sort!</button>');
        return this;
    },

    toggleSort: function (e) {
        this.fire('toggleSort', e.target.get('text'));
    }
});

Y.Base.mix(Y.DataTable, [SomeSortExt]);

new Y.DataTable({headerView: CustomHeader});

It would be odd to implement sorting in an extension when ModelList already has extremely efficient insertion-time sorting (and less efficient but still fast re-sorting) functionality built in. An extension might listen for UI events and whatnot, but ModelList should do the actual sorting.

@rgrove Yeah, I just took what @lsmith was saying as a "for instance…".

I also feel that your Views should add the DT Widget as a bubble target.

Yep, been that way since the beginning.
https://github.com/lsmith/yui3/blob/dtxpr/src/datatable/js/core.js#L324

Custom events. I'm a big believer that the View should translate DOM events, which resulted from user interactions, into higher-level custom events; in this case custom events which make sense in the world of DataTable.

The display order of the rows can be considered a view-only concern (lsmith/yui3#7). John Lindal raised this point a while back, and I agree on an engineering level. All the same, just sorting the DT's ML would be a lot easier, and probably less fragile.

Your code example reinforces my point about the use of View's events config. In order to use it, you've defined a new View subclass that replaces the current headerView. I do want the interaction to be decoupled, and using events to relay info between the DT and views is good, but why would the view fire a sort event if it didn't have any associated action? It will sort in response to a change in its modelList. If the DT will sort the ML, that should be the defaultFn of a sort event fired from the DT. If the sorting is kept a view concern, then... things get even more complicated I think. And how would you make the current HeaderView capable of firing a sort event? That's why I'm thinking plugins, but (circling back) that doesn't use the View's events config.

Hi, might be a bit offtopic here, but...

I'm trying to re-implement this jQuery example in YUI 3:

http://www.datatables.net/release-datatables/examples/basic_init/zero_config.html

And most of the functionality I've coded works, but I can't for the life of me see how to filter the DT data. Would it be possible to have something like:

Y.DataTable.Filter(pattern,fields);

Which applies filtering pattern "pattern" on all specified fields (and if fields are left empty it means "all of them")?

@perekstrom An example of the current implementation of row filtering in 3.4.1 is here: http://yuilibrary.com/yui/docs/recordset/recordset-filter.html

Though it doesn't show its direct application in DT, it's possible to use the plugin with the Recordset that DataTable uses.

This will change in 3.5.0. The migration (and augmentation) of this feature is being tracked here: lsmith/yui3#10

By all means, add suggestions there.

Just listened to today's Open Hours 1/12/2012 and looks like things are going well for the re-factoring. A few questions / comments that I wondered about during the presentation;

1. Regarding the concept of "Data Types" that was mentioned in today's Open Hours (and recalling Satyam advocating back in Aug 2010 Open Hours ... I think), we are VERY SUPPORTIVE of the idea to allow the concept of a "Data Type" (not to be confused with DataType Utility!).

We frequently use a "Data Type" custom object to enforce preferred business logic (maybe locale-specific);

 a.  Parsers ( "sql-date", "currency", "numeric", ...)
 b.  Formatters ( "currency2", "comma2", "short-date", "link", "email", ...)
 c.  Default values ( "None", "nbsp;", "No entry", ...)
 d.  Sorting ( "numeric", "date", ...)
 e.  Form INPUT display focus/blur ... (e.g. _onblur show INPUT as currency2, onfocus show INPUT as numeric2_ )
 f.  Form Validation ( "numeric", "string", "date" ), 
 g.  Save template back to Server (e.g. _JS Date conversion back to SQL formatted date onSubmit_ )

2. With respect to Sorting, the forum gets a ton of questions where people want to "unsort" after a sort, or revert a sort back one level.

• I've seen this done elsewhere (no reference!) on the UI where next to the up/down caret symbol an "x" symbol is provided.
• Not sure about coding this, (wouldn't really want to have to replicate the prior sorted keys, data bloat !!), but maybe something like sortOptions : { revertable:true } for those crazies who need to do this .

3. How would multiple TBODY be used within a bodyView, or are you keeping flexibility open for the future ?
4. We like the idea of the column being a "free-format" Object, instead of a tightly bound Attribute !
5. For the mutability methods ( addColumn, addRow, addRows ) will there be an option to specify an index, or will they always be pushed to the bottom ?
6. Thinking towards the future, since the "record" is now a Model, would it be possible / feasible to;

• use Model's .save() method to send data BACK to a Server (... similar to the asyncSubmitter concepts in days of old)?
• likewise (... going out on a crazy pedantic limb here), could Model's sync( 'update' ) potentially be wired to a WebSocket to update DataTable INCOMING (i.e. "unrequested") via a push from the remote Server (i.e. ... thinking of inventory updating, stock price updating, etc...).

@stlsmiths

re: 2 - I'm not planning on supporting a configuration for it, but you should be able to call table.sort('clientId') to resort by insertion order. That should DWYW.

re: 3 - This was a concession for the sake of time. It may come back to bite me in the ass, but I abandoned the notion of having the views generate their own relative containers (see https://gist.github.com/1356355#gistcomment-62759) because it was more convenient to have the important Nodes as properties on the DT instance (e.g. table._theadNode) but I didn't want to have to deal with a this._tbodies collection when in the majority of cases, there would be only one. It simplified all code that dealt with the data rows to have a single Node property for the data container, and _tbodyNode matched _tfootNode and _theadNode. If a custom bodyView wants to render multiple <tbody>s, it can hack the addition of more somewhere in the table's rendering lifecycle. Ugly, I admit.

re: 5 - row insertion point is managed by ModelList's sorting algo, but addColumn does accept an insertion index. - http://lsmith.github.com/yui3-gallery/api/classes/DataTable.html#method_addColumn

re: 6 - Yes. Party at my house :)