First and foremost, I want to say that the idea of reactive queries vs subscriptions is a great direction and one that greatly simplifies the syntax and process of getting data. After some experimentation with Falcor and Relay, it certainly seems like different approach is needed. After doing a fare amount of work both Redux and with SSR for Meteor, simplifying the data process is a must.
The client and server parts of the GraphQL system will need to collaborate to make this possible, but one of our goals is to make minimal changes to the current reference implementation of GraphQL so that developers can take advantage of current and future productivity tools for GraphQL.
I think this is critical to this project. I know its in the vein of this project but I've seen our team bounce around from abstraction libraries that obscure the underpinnings of what is going on and its very frustrating to break from spec. So all in all, I appreciate this approach and want to highlight its importance.
{
me: {
username: "sashko",
lists: [
{
id: 1,
name: "My first todo list",
tasks: [ ... and so on ],
_deps: {
_self: { key: '12341234', version: 3 },
tasks: { key: '35232345', version: 4 }
}
}
]
}
}
This is a minor thing, but considering GraphQL already supports metadata of sorts via __schema and __type, would it be better to have a __deps and __self fields instead of _deps and _self?
the automatic dependency recording mechanism won’t work if it isn’t possible to analyze the query to determine the dependencies. In these cases, the developer will need to manually record a dependency, using any string key they want.
This will be particularly useful when querying REST endpoints or even working with webhooks.
The client fetches the query from the GraphQL server, which includes a set of deps in the response.
I think it would be useful to strip the deps on the client cache side before returning the data to the fetch point. One of the core tenants (imo) of GQL is the declarative nature. If a developer / designer specifies a query, and plans to iterate over the object expecting a limited set of keys, it could be frustrating to find things you didn't call for. I know relay returns id but in some cases even this could be an anti pattern given GQL's assignable nature. Take for instance
// fetching a person for use in showing who is online
{
people(limit: 10) {
id: username, // any unique field can be used as the id
firstName,
lastName
}
}
The above may seem odd but when a designer is iterating over and using uniqueness as a decider for things, in some cases the id may be wrong due to it being sensitive data not wanting to expose to the client, or it may not make the most sense for the data model being declared on the client.
The invalidation server could accept websocket connections, and let the client subscribe to the dependency keys it cares about - this would mean the invalidation is pushed immediately, and then there is one roundtrip to fetch the actual data
Our current application scope extends to a *:1 relationship between client applications (not client connections, but many applications (native apps, web apps, IOT devices)) and our centralized GQL server for our organization. As we plan to scale up to even a SaaS model for other organizations, we want to be able to scale out our GQL servers to a large amount and not be as concerned about mapping client sessions to a specific server. All that being said to this point, I think having the invalidation server live in memory on a single instance might not scale well, whereas having it store the deps in a redis or redis like db that all horizontally scaled instances of our GQL server pulls from is ideal. We run into this with IIS Sessions on some of our ASP.NET projects. Then where websockets make sense, have a 1:* GQL:client connections with session persistence to update clients makes great sense, but other applications can poll, not care about which server they hit because they would all be pulling deps from a shared key:value store.
Hopefully, with time, we can make more and more invalidations automatic, but it’s always good to have an “escape hatch” for more complex situations where the developer needs all the control they can get.
Making this great is easily one of the hardest issues from what we have run into. We pull data from multiple dbs that differnt applications maintain. Think a CMS, a RMS (Relational Management System e.g hubspot, infusionsoft, etc), and our application db. We currently try to pipe all mutations to the application to handle data updating. Ideally, and DB support not with standing, I'd like to track tables / collections and fields within them for changes to invalidate the deps, instead of on the upsert/update. That way 3rd party applications still own data changes, and we listen for updates when possible.
However, we expect more friendly data drivers to be written by the community, in addition to the official ones maintained by Meteor, which may include one or more of SQL, MongoDB, and REST APIs.
We are working on early versions of this for MySQL, MSSQL, MongoDB, and webhook based API fwiw. We probably won't have anything solid for a few more months as we are launching a few applications to production next month, but it is a high priority for my team.
Have basic caching for better performance
I think this also extends to the cache lookup. Facebooks DataLoader looks particularly interesting for batching queries along process ticks.
support performance analysis both for development and production use
Having these hooks allow for middleware would be huge for integration into systems like pagerduty, newrelic, raygun, etc.
How a specific set of UI components on the page translates into a GraphQL query
I think this is part of a larger discussion, but we have moved all of our queries to a mix of non UI container components as well as using redux based sagas handle data fetching. We are trying to decouple our UI from data as much as possible and use routes / containers to fetch the needed data and pipe it to UI components. I'd love this system to allow for something similar. As we dual between client and server based data aggregation, having the client perform background data fetching to pre-render / pre-load has been really great for us. We have fail-safes on UI containers that have to have certain data, but we share a decent amount between routes / components so after the client has been server side rendered and the application has been instantiated correctly, we start fetching background data that we think the client will need based on possible action points on a page / app. It is like optimistic UI but with optimistic data.
Application Server(s)
I think an easy way to secure our application server (using bearer tokens, Meteor auth, JWT, or more) would be a great benefit to this system. Auth always seems harder than it needs to be in my experience. (Maybe start as a Meteor add on, then add other methods?)
I think wherever the client connection ultimately interfaces with the GQL server (we currently go client -> app server (for auth) -> GQL server), an easy way to integrate caching is important. However I think this could be a shared role between the DB libraries and the db tracking context. So the Express app would go Query -> Dep (as cache) -> db. And the reactive db bindings would update the Dep context (as you outlined here)
We plan on MongoDB, MSSQL, MySQL (maybe Amazon Aurora as well), and WebSockets as part of our work this year. Eventually we want to integrate an actual GraphDB (like Neo4j) as well.
Invalidation Server
We have been toying around with running this in a forked process as a background task but writing it as a Golang binary that an npm package wraps. Personally, I'd like to see these two (GQL and Invalidation server) live together so they don't require multiple containers / servers. I'd rather see the client have a single connection point to secure than multiple.
Caching and Network Interaction
This looks great overall. I'm interested to know in what context we can utilize service workers to enhance this store. Mainly due to its offline support and per the spec
Service worker is a programmable network proxy, allowing you to control how network requests from your page are handled.
Could be a good enhancement?
Just as a personal aside, I've been writing a lot of generators recently using redux-saga to handle async actions and remote data interactions and they have been a joy to write. Its probably how we are going to start our work on the mutation wrapper / invalidation store actions.
What are your thoughts on the store model? Keeping it as normalized as possible seems like a win, especially for shared data points (e.g. lists shared between to owners)
We would probably end up writing a redux tie in fwiw so we can use our redux patterns (sagas, reducers, etc) if they aren't replicated in this system.
Query Aggreation and Data Injection
I'd like to volunteer our team to work on React Router integration as it is what we use for our apps and would be doing it anyway :).
All in all I'm a huge fan of this, and it is what my team is planning on tackling a lot this year for our sites and applications anyway. A robust CMS solution with GQL and react + meteor has kept us from moving as fast as we want for a while, and I think this is a great medium for having that work great.
I also really appreciate the openness and willingness to let us (community) be a part of this project. I'm looking forward to seeing how we can move it together!
Thanks so much @stubailo