hyperfiddle-metamodel.md

This is probably going to be the next iteration of the declarative CRUD metamodel that powers Hyperfiddle. It's just a design sketch, the current metamodel in prod is different. Hyperfiddle is an easy way to make a CRUD app. Hyperfiddle is based on Datomic, a simple graph database with the goal of "enabling declarative data programming in applications."

CRUD UI definition

This extends Datomic pull syntax into something that composes in richer ways. The key idea is that Pull notation expresses implicit joins and thus can be used to declare data dependencies implicitly, without needing to name them. We also handle tempids, named transactions, and hyperlinks to other pages. We satisfy the hypermedia constraint, like HTML and the web.

{identity                                                   ; Pass through URL params to query
 [{:dustingetz/event-registration                           ; virtual attribute identifying a query
   [:db/id
    (:dustingetz/email {:hf/a :dustingetz/registrant-edit}) ; hyperlink to detail form
    :dustingetz/name
    {:dustingetz/gender
     [:db/ident]}
    {:dustingetz/shirt-size
     [:db/ident]}]}]

 nil                                                        ; no query params
 [{:dustingetz/genders                                      ; genders query (for picklist)
   [:db/ident]}]

 ((hf/new) {:hf/tx :dustingetz/register})                   ; generate a Datomic tempid and wire up a transaction
 [:db/id
  :dustingetz/email
  :dustingetz/name
  {:dustingetz/gender
   [:db/ident
    {:dustingetz/shirt-sizes                                ; shirt-sizes query depends on gender
     [:db/ident]}]}
  {:dustingetz/shirt-size
   [:db/ident]}]}

We hesitated to go in the direction of extending Datomic syntax because it could conflict in the future, however Rich has been quite clear that Datomic queries are data, maps should be open (explicitly should not be closed), etc, and this is all safe because data is namespaced. The benefit to this syntax is it's simple enough that it doesn't require tool assistance to write (the current hyperfiddle metamodel is not palatable without UI assistance).

The popover is triggered by (hf/new) which allocates a tempid for a CREATE operation. CREATE operations are composed of a parent query and a child form (implicit in the pull). The popover indicates two things: 1) where in the graph the parent/child reference is, and 2) that the insertion is transactional and can be discared as a unit.

Queries

Queries are declarative which lets us use metaprogramming techniques to manipulate them as data, for example a client UI typeahead picker may specify additional database filters as datalog which will be spliced into the query.

{:dustingetz/genders
 [:find (pull ?e [:db/ident])
  :where
  [?e :db/ident ?i]
  [(namespace ?i) ?ns]
  [(ground "dustingetz.gender") ?ns]]

 :dustingetz/shirt-sizes
 [:in $ ?gender                                             ; Query dependency, automatically inferred from the pull
  :find (pull ?e [:db/ident])
  :where
  [?e :db/ident ?i]
  [?e :dustingetz.reg/gender ?gender]
  [(namespace ?i) ?ns]
  [(ground "dustingetz.shirt-size") ?ns]]

 :dustingetz/entity-history                                 ; just an example of how server code eval might work
 (->> (d/q '[:in $ ?e
             :find ?a ?v ?tx ?x
             :where
             [?e ?a ?v ?tx ?x]]
           (d/history *$*)
           %)
      (map #(assoc % 0 (:db/ident (d/entity db (get % 0)))))
      (group-by #(nth % 2))
      (map-values #(sort-by first %))
      (sort-by first))}

Validation

This is spec1, we're on a collision course with spec2 and looking forward to discovering how they unify.

(s/def :dustingetz/register (s/keys :req [:dustingetz/email
                                          :dustingetz/name]
                                    :opt [:dustingetz/gender
                                          :dustingetz/shirt-size]))

View progressive enhancement

Hyperfiddle handles forms & tables automatically and lets you customize rendering on an attribute basis. You can of course control the entire renderer. Picklists are simply view progressive enhancement of a form field plus a query. Note that there is no data-sync I/O, async, error handling or other side effects in views!

(defmethod hyperfiddle.api/render #{:dustingetz/gender}     ; custom renderer for ::gender
  [ctx props]
  [hfui/select ctx                                          ; present as picklist
   {:options :dustingetz/genders                            ; Wire up picklist options to named query
    :option-label :db/ident}])                              ; picklist label

(defmethod hyperfiddle.api/render #{:dustingetz/shirt-size} ; custom renderer for ::shirt-size
  [ctx props]
  [hfui/select ctx                                          ; second picklist
   {:options :dustingetz/shirt-sizes
    :option-label :db/ident
    :hf/where '[[(name ?i) ?name]                           ; client specified database filters
                [(clojure.string/includes? ?name %)]]}])

Transactions

Transactions are attached to the stage button of popovers. They are positioned at a point in the graph through the pull, so their EAV parameters can be inferred.

(defmethod hyperfiddle.api/tx :dustingetz/register [ctx [e a v] props]
  [[:db/add v :dustingetz/registered-date (js/Date.)]])

This method runs and it's return value is concatenated with the popover form datoms, for final form submission in a single transaction. If you click cancel on a popover it discards the form datoms without submitting any transaction. The form value is available for inspection via ctx as well as any other query or value in scope whose dependencies are satisfied (e.g. ::shirt-sizes is available because there is a ::gender in scope).

In a prod configuration, transaction functions evaluate securely on the server (behind an HTTP POST or something).

Datomic Schema

Hyperfiddle understands: ident, valueType, cardinality, unique, isComponent and generates idiomatic Datomic transactions, including lookup refs.

[{:db/ident :dustingetz/name,
  :db/valueType :db.type/string,
  :db/cardinality :db.cardinality/one,
  :db/doc "Registrant's name"}
 {:db/ident :dustingetz/email,
  :db/valueType :db.type/string,
  :db/cardinality :db.cardinality/one,
  :db/unique :db.unique/identity,
  :db/doc "Registrant's email"}
 {:db/ident :dustingetz/gender,
  :db/valueType :db.type/ref,
  :db/cardinality :db.cardinality/one,
  :db/doc "Registrant's gender (for shirt size)"}
 {:db/ident :dustingetz/shirt-size,
  :db/valueType :db.type/ref,
  :db/cardinality :db.cardinality/one,
  :db/doc "Selected tee-shirt size"}]

From such a metamodel you could generate not just CRUD UI, but also data-sync, APIs, adapters for foreign systems, intelligent systems, optimizing infrastructure, schema-stitching, cross domain joins, cross-company data partnerships...

http://hyperfiddle.net

{identity                                                   ; Pass through URL params to query
 [{:dustingetz/event-registration                           ; virtual attribute identifying a query
   [:db/id
    (:dustingetz/email {:hf/a :dustingetz/registrant-edit}) ; hyperlink to detail form
    :dustingetz/name
    {:dustingetz/gender
     [:db/ident]}
    {:dustingetz/shirt-size
     [:db/ident]}]}]

 (constantly nil)                                                        ; no query params
 [{:dustingetz/genders                                      ; genders query (for picklist)
   [:db/ident]}]

 ((hf/new) {:hf/tx :dustingetz/register})                   ; generate a Datomic tempid and wire up a transaction
 [:db/id
  :dustingetz/email
  :dustingetz/name
  {:dustingetz/gender
   [:db/ident
    {:dustingetz/shirt-sizes                                ; shirt-sizes query depends on gender
     [:db/ident]}]}
  {:dustingetz/shirt-size
   [:db/ident]}]}


(datomic/pull
  [:db/id
   :dustingetz/email
   :dustingetz/name
   {:dustingetz/gender
    [:db/ident
     {:dustingetz/shirt-sizes
      [:db/ident]}]}
   {:dustingetz/shirt-size
    [:db/ident]}])



(hf.api/q
  [{(juxt identity identity)
    [:db/id
     :my-query
     `my-query
     (constantly nil)]

    identity
    {:db/id :my-query}
    }])

{:db/id 123
 :my-query 123
 `my-query 123
 `(constantly nil) 123
 `c-nil}

(ns user)
(defn c-nil []
  nil)

(defn my-query [e]
  (datomic.api/entity e :db/id))

(defquery :my-query [e]
  (datomic.api/entity e :db/id))

(ns scratch
  (:require
    [datomic.api :as d]
    [backtick :refer [template]]
    [hyperfiddle.config]
    [hyperfiddle.domain]
    [hyperfiddle.api :as hf]
    ))

(comment
  (def config (hyperfiddle.config/get-config "./hyperfiddle.edn"))
  (def domain (hyperfiddle.config/get-domain config))
  (def conn (hyperfiddle.domain/connect domain "$"))
  (def $ (d/db conn))
  (def $ (:db-after (d/with $ [
                               {:db/ident ::email :db/valueType :db.type/string :db/cardinality :db.cardinality/one}
                               {:db/ident ::gender :db/valueType :db.type/ref :db/cardinality :db.cardinality/one}
                               {:db/ident ::shirt-size :db/valueType :db.type/ref :db/cardinality :db.cardinality/one}
                               {:db/ident ::type :db/valueType :db.type/keyword :db/cardinality :db.cardinality/one}
                               ])))
  (def $ (:db-after (d/with $ [{::email "[email protected]"}
                               {::type ::gender :db/ident ::male}
                               {::type ::gender :db/ident ::female}
                               ])))
  (def $ (:db-after (d/with $ [{::type ::shirt-size :db/ident ::mens-small ::gender ::male}
                               {::type ::shirt-size :db/ident ::mens-medium ::gender ::male}
                               {::type ::shirt-size :db/ident ::mens-large ::gender ::male}
                               {::type ::shirt-size :db/ident ::womens-small ::gender ::female}
                               {::type ::shirt-size :db/ident ::womens-medium ::gender ::female}
                               {::type ::shirt-size :db/ident ::womens-large ::gender ::female}
                               ])))

  )

; These functions are essential, they can't be changed
; because we need the full Datomic API plus clojure/core

(defn submissions [pp $]
  (let [qq (template
             [:find (pull ?e ~pp)
              :where [?e ::email]])]
    (d/q qq $)))

(defn genders [pp $]
  (d/q (template [:find [(pull ?e ~pp) ...] :where [?e ::type ::gender]]) $))

(defn shirt-sizes [pp $ gender]
  (d/q (template [:in $ ?gender
                  :find [(pull ?e ~pp) ...]
                  :where
                  [?e ::type ::shirt-size]
                  [?e ::gender ?gender]])
    $ gender))



(comment
  (submissions [::email
                {::gender [:db/ident]}
                {::shirt-size [:db/ident]}]
    $)
  (genders [:db/ident] $)
  (shirt-sizes [:db/ident {::gender [:db/ident]}] $ ::male)

  )

; Goal
; Model the submissions table, also loading it's select options for both genders and shirt-sizes as needed

[{`submissions [::email
                {::gender [:db/ident
                           {`shirt-sizes [:db/ident]}]}
                {::shirt-size [:db/ident]}]}
 {`genders [:db/ident]}]

; How does the UI connect the select renderer to the options picklist?

(defmethod hf/render #{::gender} [ctx props]
  [select ctx {:options `genders
               :option-label (comp str :database/uri)}])

(defmethod hf/render #{::shirt-size} [ctx props]
  [select ctx {:options `shirt-sizes                        ; resolves the gender based on table row in render ctx
               :option-label (comp str :database/uri)}])