benkolera · January 31, 2019 00:01
diff --git a/gistfile1.txt b/gistfile1.txt
 jbetz> is `rhyolite` for enabling the frontend to subscribe to backend events via web sockets? and has anyone outside of obsidian tried using it?
 02:06 <Cale> jbetz: rhyolite is a bunch of partially-baked miscellaneous libraries which are at various levels of polish, but which we needed to open-source early for various reasons. Use it at your own risk, but please let us know if you find anything particularly good/bad if you do end up trying it out.
 02:07 We've been breaking out pieces of it, taking off the sharp edges, and open sourcing them as separate libraries as time permits.
 02:08 Rhyolite.Backend.App does contain something like what you described though
 02:10 → mtesseract joined ([email protected])
 02:13 <jbetz> Cale: gotcha. I got curious when I started looking through obsidian's tezos projects.
 02:18 also, I really like the idea of widget's subscribing to database queries. there's a clojure project called `precept` that pretty much does this. the database is just a graph of facts, widgets update it by inserting new ones, and then you have a bunch of rules that determine how the graph gets updated. so basically what you end up with is a prolog rules engine.
 02:22 <Cale> Basically, our frontends collect up, as a Dynamic, information about what all the sub-widgets are interested in using QueryT from Reflex, in a structure we generically refer to as a view selector. It's got to be a monoid (to combine the requests throughout the frontend), and actually a group (as an optimisation, we add/subtract things rather than re-combining everything when stuff changes)
 02:22 <jbetz> I've been trying to layer something like this on top of `reflex` as a proof of concept. I think it's possible, though not very idiomatic, and definitely not performant.
 02:22 <Cale> and then the backend responds with a "view", which again has some monoid structure (typically both of these things are records of Maps)
 02:23 and in addition to that, it takes everyone's view selectors, and combines them together into a view selector that records the set of users interested in each individual piece of information
 02:24 and then as the database is updated, we use postgres NOTIFY/LISTEN to let our backends know about what's going on, and they take those updates and distribute new partial views to the interested users accordingly
 02:25 ⇐ mtesseract quit ([email protected]) Ping timeout: 244 seconds
 02:26 <Cale> We have some unfinished stuff in the works for consuming postgres' write log directly, rather than having to effectively use the DB as an IRC server to tell ourselves when stuff is being updated, but we're not even using that for internal projects yet.
 02:28 I also have open sourced https://github.com/obsidiansystems/vessel which, though we are not yet using it internally, is going to be our common data structure for both views and view selectors in the future. It's a fancy sort of Map-like container, with the following properties:
 02:30 It is parameterised by a choice of Functor, which is applied to all the "leaves" of the datastructure: basically, by plugging in Proxy, we get a frontend view selector, by plugging in Identity, we get a view, by plugging in something like Map, we get the sort of aggregate views and view selectors that the backend has to deal with
 02:30 and then the keys of a vessel are elements of some GADT which is indexed by such Functor-parametrised containers, which say which sort of container is stored alongside that key
 02:31 So you can "nest" Vessels, and there are other functor-parametric containers for ordinary Map structures and other stuff that could go in there
 02:32 The idea being that we eventually want to be able to take entire application protocols and be able to nest them together and reuse whole sections of an app.
 02:33 <jbetz> fascinating
 02:33 <Cale> A lot of what was annoying about our existing view/selector approach is the tremendous amount of boilerplate instances you'd have to write, and Vessel automates most of that
 02:34 (it just has all the required Group instances, and things related to the transposition that needs to go on to aggregate users' view selectors, and crop views to what was selected by a selector, etc.)
 02:34 and JSON, etc. et.c
 02:35 Writing the JSON instances for Vessel was really interesting actually
 02:35 because the constraints you need on that instance are very unusual
 02:36 You need to be able to say: for any possible *value* of the key type, there is a JSON instance for the corresponding container type that its type index tells you.
 02:36 So it's like constraint-Pi-types
 02:36 https://github.com/obsidiansystems/constraints-extras -- so we ended up with this stuff
 02:37 In addition to that, of course, you also need a way to generate JSON instances for your keys
 02:37 which are GADTs, so the usual aeson generics don't work
 02:37 https://github.com/obsidiansystems/aeson-gadt-th/ -- so we did this :D
 02:38 <jbetz> lol
 02:40 > 

    combines them together into a view selector that records the set of users interested in each individual piece of information <-- if each piece of information is a topic, this is basically a pub-sub system? or at least, the sub part of it. is there any sort of symmetrical functionality for the pub part?

 02:40 <lambdabot> Lambda_Robots:_100%_Loyal  <hint>:1:66: error: parse error on input ‘of’
 02:43 <Cale> Yeah, so the way we'd been doing it up until Vessel is that both the view and view selector types are functors
 02:44 and each piece of the query or corresponding piece of information in a view will be paired up with an arbitrary value of some type
 02:44 On the frontend, that's basically Integer (reference counting how many widgets are interested)
 02:44 for the selector
 02:44 On the frontend for the view, that's just ()
 02:45 On the backend, those become sets of connected clients who are interested in that information
 02:46 (well, in the part which handles notification about updates anyway)
 02:47 There's nothing which actually *enforces* a connection between the view and selector in the old way of doing things, apart from a hand-written crop function which, if you think of the view as a partial copy of your database, is effectively the same thing as the SQL you'll write to obtain a view from the selector. (except annoyingly, you have to write it again, in Haskell)
 02:48 I got a bit sick of that, and so with Vessel, you're simply required to have things correspond structurally -- it's a little bit limiting in some ways, but I think it'll end up being fine.
 02:49 So then the cropping can be done using what is effectively a (recursive) Map intersection.
 03:17 ⇐ ryantrinkle quit ([email protected]) Ping timeout: 246 seconds
 03:22 <jbetz> what are the use cases for QueryT? is it only for when you want incremental updates to a Dynamic, or are there others?
 03:28 <Cale> As far as I'm aware, it's basically exactly that sort of scenario, where you want to collect up requests for information, and then distribute it back out through your application.
 03:30 https://github.com/reflex-frp/reflex/blob/develop/src/Reflex/Query/Class.hs#L35 -- you'll see that there's a crop function you specify there, which will be applied to the overall response at each of the places where you're requesting information
 03:31 https://github.com/reflex-frp/reflex/blob/develop/src/Reflex/Query/Class.hs#L96 -- queryDyn is the main thing you'll use to actually make queries (in fact, it's mostly just that and runQueryT, all this other stuff is rather special-purpose or implementation details)
 03:33 So you give it a Dynamic t q, where q is your query type, and run it, and it gives you back a Dynamic t (QueryResult q), which is the global query result that got provided by runQueryT, cropped according to the query provided there
 03:36 → ryantrinkle joined ([email protected])
 03:43 <jbetz> hmm, okay
 03:43 I thought `crop` would be a sort of lens on `a`, that specifies which subpart of the dynamic you're interested in
 03:44 s/a/q
 03:47 the `MonoidalMap` explains it though, since cropping could just be removing associations
 03:51 can you stack QueryT such that it's possible to run multiple types of queries within the same monad, and express which ones via constraints?
 04:14 → Gurkenglas joined (~Gurkengla@unaffiliated/gurkenglas)
 04:31 <jbetz> I guess that really isn't necessary since `QueryResult a` has to be a semigroup, so you can always just combine them
	jbetz> is `rhyolite` for enabling the frontend to subscribe to backend events via web sockets? and has anyone outside of obsidian tried using it?
	02:06 <Cale> jbetz: rhyolite is a bunch of partially-baked miscellaneous libraries which are at various levels of polish, but which we needed to open-source early for various reasons. Use it at your own risk, but please let us know if you find anything particularly good/bad if you do end up trying it out.
	02:07 We've been breaking out pieces of it, taking off the sharp edges, and open sourcing them as separate libraries as time permits.
	02:08 Rhyolite.Backend.App does contain something like what you described though
	02:10 → mtesseract joined ([email protected])
	02:13 <jbetz> Cale: gotcha. I got curious when I started looking through obsidian's tezos projects.
	02:18 also, I really like the idea of widget's subscribing to database queries. there's a clojure project called `precept` that pretty much does this. the database is just a graph of facts, widgets update it by inserting new ones, and then you have a bunch of rules that determine how the graph gets updated. so basically what you end up with is a prolog rules engine.
	02:22 <Cale> Basically, our frontends collect up, as a Dynamic, information about what all the sub-widgets are interested in using QueryT from Reflex, in a structure we generically refer to as a view selector. It's got to be a monoid (to combine the requests throughout the frontend), and actually a group (as an optimisation, we add/subtract things rather than re-combining everything when stuff changes)
	02:22 <jbetz> I've been trying to layer something like this on top of `reflex` as a proof of concept. I think it's possible, though not very idiomatic, and definitely not performant.
	02:22 <Cale> and then the backend responds with a "view", which again has some monoid structure (typically both of these things are records of Maps)
	02:23 and in addition to that, it takes everyone's view selectors, and combines them together into a view selector that records the set of users interested in each individual piece of information
	02:24 and then as the database is updated, we use postgres NOTIFY/LISTEN to let our backends know about what's going on, and they take those updates and distribute new partial views to the interested users accordingly
	02:25 ⇐ mtesseract quit ([email protected]) Ping timeout: 244 seconds
	02:26 <Cale> We have some unfinished stuff in the works for consuming postgres' write log directly, rather than having to effectively use the DB as an IRC server to tell ourselves when stuff is being updated, but we're not even using that for internal projects yet.
	02:28 I also have open sourced https://github.com/obsidiansystems/vessel which, though we are not yet using it internally, is going to be our common data structure for both views and view selectors in the future. It's a fancy sort of Map-like container, with the following properties:
	02:30 It is parameterised by a choice of Functor, which is applied to all the "leaves" of the datastructure: basically, by plugging in Proxy, we get a frontend view selector, by plugging in Identity, we get a view, by plugging in something like Map, we get the sort of aggregate views and view selectors that the backend has to deal with
	02:30 and then the keys of a vessel are elements of some GADT which is indexed by such Functor-parametrised containers, which say which sort of container is stored alongside that key
	02:31 So you can "nest" Vessels, and there are other functor-parametric containers for ordinary Map structures and other stuff that could go in there
	02:32 The idea being that we eventually want to be able to take entire application protocols and be able to nest them together and reuse whole sections of an app.
	02:33 <jbetz> fascinating
	02:33 <Cale> A lot of what was annoying about our existing view/selector approach is the tremendous amount of boilerplate instances you'd have to write, and Vessel automates most of that
	02:34 (it just has all the required Group instances, and things related to the transposition that needs to go on to aggregate users' view selectors, and crop views to what was selected by a selector, etc.)
	02:34 and JSON, etc. et.c
	02:35 Writing the JSON instances for Vessel was really interesting actually
	02:35 because the constraints you need on that instance are very unusual
	02:36 You need to be able to say: for any possible value of the key type, there is a JSON instance for the corresponding container type that its type index tells you.
	02:36 So it's like constraint-Pi-types
	02:36 https://github.com/obsidiansystems/constraints-extras -- so we ended up with this stuff
	02:37 In addition to that, of course, you also need a way to generate JSON instances for your keys
	02:37 which are GADTs, so the usual aeson generics don't work
	02:37 https://github.com/obsidiansystems/aeson-gadt-th/ -- so we did this :D
	02:38 <jbetz> lol
	02:40 >

	combines them together into a view selector that records the set of users interested in each individual piece of information <-- if each piece of information is a topic, this is basically a pub-sub system? or at least, the sub part of it. is there any sort of symmetrical functionality for the pub part?

	02:40 <lambdabot> Lambda_Robots:_100%_Loyal <hint>:1:66: error: parse error on input ‘of’
	02:43 <Cale> Yeah, so the way we'd been doing it up until Vessel is that both the view and view selector types are functors
	02:44 and each piece of the query or corresponding piece of information in a view will be paired up with an arbitrary value of some type
	02:44 On the frontend, that's basically Integer (reference counting how many widgets are interested)
	02:44 for the selector
	02:44 On the frontend for the view, that's just ()
	02:45 On the backend, those become sets of connected clients who are interested in that information
	02:46 (well, in the part which handles notification about updates anyway)
	02:47 There's nothing which actually enforces a connection between the view and selector in the old way of doing things, apart from a hand-written crop function which, if you think of the view as a partial copy of your database, is effectively the same thing as the SQL you'll write to obtain a view from the selector. (except annoyingly, you have to write it again, in Haskell)
	02:48 I got a bit sick of that, and so with Vessel, you're simply required to have things correspond structurally -- it's a little bit limiting in some ways, but I think it'll end up being fine.
	02:49 So then the cropping can be done using what is effectively a (recursive) Map intersection.
	03:17 ⇐ ryantrinkle quit ([email protected]) Ping timeout: 246 seconds
	03:22 <jbetz> what are the use cases for QueryT? is it only for when you want incremental updates to a Dynamic, or are there others?
	03:28 <Cale> As far as I'm aware, it's basically exactly that sort of scenario, where you want to collect up requests for information, and then distribute it back out through your application.
	03:30 https://github.com/reflex-frp/reflex/blob/develop/src/Reflex/Query/Class.hs#L35 -- you'll see that there's a crop function you specify there, which will be applied to the overall response at each of the places where you're requesting information
	03:31 https://github.com/reflex-frp/reflex/blob/develop/src/Reflex/Query/Class.hs#L96 -- queryDyn is the main thing you'll use to actually make queries (in fact, it's mostly just that and runQueryT, all this other stuff is rather special-purpose or implementation details)
	03:33 So you give it a Dynamic t q, where q is your query type, and run it, and it gives you back a Dynamic t (QueryResult q), which is the global query result that got provided by runQueryT, cropped according to the query provided there
	03:36 → ryantrinkle joined ([email protected])
	03:43 <jbetz> hmm, okay
	03:43 I thought `crop` would be a sort of lens on `a`, that specifies which subpart of the dynamic you're interested in
	03:44 s/a/q
	03:47 the `MonoidalMap` explains it though, since cropping could just be removing associations
	03:51 can you stack QueryT such that it's possible to run multiple types of queries within the same monad, and express which ones via constraints?
	04:14 → Gurkenglas joined (~Gurkengla@unaffiliated/gurkenglas)
	04:31 <jbetz> I guess that really isn't necessary since `QueryResult a` has to be a semigroup, so you can always just combine them