calculang - update

a) readme update?

calculang is a language for calculations.

(Note: I say modelling and calculation language interchangably.)

Why?

- separation of concerns

  We can calculate in any programming language: and this is part of the problem calculang considers to address.
  e.g. we can't follow calculation logic across arbitrary systems and languages, and we're limited wrt modelling-specific tooling and reasoning we can apply.
  
  A standard language for calculations which can be used by applications and systems as-needed is the core of calculangs technical proposal.

- unity

  In calculang, we **can** talk about following calculation logic across calculang models with **minimal friction**.
  
  This is important, because no number is an island: the input data to one model is the output data from another model. "web" and "fluid"

- transparency

  Numbers have stakeholders, e.g. customers/prospective customers, colleagues, bosses, readers.
  We do not strictly **need** a standard language for calculations to provide transparency to stakeholders: we can publish Excel spreadsheets, Python/R/matlab source code, etc.
  However, a standard language for calculations can **minimise the friction** and empower specialised, **more intuitive user interfaces**.
  
- democratisation of numbers

  Numbers affect people: bank and loan balances, spending decisions, bills, salary, purchase/sales prices, household budgets, bonuses, taxes, benefits, credit ratings, pensions, government budgets, government policy, investment decisions, scientific publications, ...
  We should have an interest with understanding, questioning, and validating numbers that are used to affect our lives or influence us and that affect our own decisions.
  At the very least, we should not feel excluded along those lines.
  So we should talk about democratisation of numbers, and calculang wants to empower all stakeholders to make this more meaningful.

Notes:

 - Top two points are technical. Bottom two points are about meaning, but are enabled by each of the top two technical points.
 
 - Users may not care about the wider meaning applied in the bottom two points, but I think they normally care about the same values locally.
   Facilitating transparency is important inside an organisation as well as going outside the bounds of an organisation.
   But "facilitating" means just that - it doesn't mean transparency is forced, and there are many cases where transparency is actually undesirable!
       
 - I think it is hard to overstate the **importance and potential impact of minimising friction**.

b) what does calculang look like?

The implementation (calculang-js) is in Javascript: so it looks like Javascript as a starting point.

But, the compiler does some work to facilitate calculang features.
Mostly, this is about minimising boilerplate and facilitating recycling of models.
("we shouldn't code the wiring between functions manually because it will differ from one usecase to the next")

examples: see https://observablehq.com/@declann/calculang-compilation-examples

calculang is also averse to constants (non-functions): in calculang a constant should be abstracted to a function returning a constant.
This is in line with facilitating recycling of models, because another usecase might want to model that constant with a more sophisticated calculation (i.e. function).
This is why model inputs - which are constants, are always abstracted to functions.

There is a useful **uniformity** then: calculang models are made up of functions, and relationships are fully characterized by calls between functions.

c) what does calculang look like? some perspectives

Programmer world:

I'm not an expert on **functional programming**, but I think calculang fits right into this space.
calculang facilitates the development of **highly-parameterised models of pure-functions**.
**Its approach aims to minimise boilerplate and maximise flexibility.**

calculang (atm) doesn't prevent you from running imperative Javascript with side-effects in your models: but doing that might be a bad idea.
I suggest side-effecty Javascript usually should be in separate modules, which you can call from calculang or vica versa,
and calls to Javascript should probably be pure in the sense that same inputs == same result.


'Real world':

This might be stretching the imagination a little but I think there is a fair comparison in language concept to Excel,
but with **references in terms of meaning instead of being attached to layout**.

Layout isn't a concept at all with calculang models: it's only relevant to a presentation of model results. This is separation of concerns.
(There is a comparison right here to how CSS was introduced to separate presentation and content on the web)

With a particular (even important) layout, calculang development need not appear very alien versus Excel/spreadsheets.
I propose that it should scale better in terms of calculations/development and be more flexible however :)



+ faceting note here?

d) status - technical

calculang compiler, @calculang/calculang-js:
 - .cul.js entrypoint -> .js bundles with .js.map sourcemap (uses webpack v4)
 - working incl. with modular models, but affected by issue #8
 - cleanup needed but probably after issue #12 ('bundler/transpiler considerations' particularly because recursive calling of webpack api used in memo-loader is unsupported)
 
model/bundle execution:
 - the execution model leaves a lot to be desired e.g. where recursiveness is deep, but also generally
 - a big boost for deep recursiveness is memoisation. I created a memo-loader PR (due in alpha 5), but implementation is only considered good for independent models that don't use modularity
 - even with memoisation, execution model will not scale well: huge technical improvements needed to scale, this is a project in itself
 - for POC, simple models, well-bounded problems, I think there is something to work with

I think the status in readme is accurate and important:

"Initial implementation scales better conceptually rather than practically, but should be useful for carefully bounded problems and a useful foundation/discussion project towards project motivations."

I will be more attached to feedback about how the concept does or doesn't scale conceptually, rather than limitations in the current implementation.

Extensions: shortcut?

e) status - documentation, examples

Examples are important to make the project goals tangible, and therefore hopefully more compelling.

I intend for my examples to be in ObservableHQ notebooks in order to move quickly. Eventually I would like to launch calculang.party to showcase models, especially from others, if that will happen.

Location of my examples: https://observablehq.com/collection/@declann/calculang

So far these 'examples' are concerned with 2 things only: engaging with a programming audience specifically, and to get old work and ideas out in public. This is debatably a decade of work, so coherency will take time.

But I do need to make the project goals tangible.

I have some specific ideas in that regard. These are important, and may be more developed apps rather than ObservableHQ notebooks:


 - a loan/mortgage validator and scenario tool: ultimately hoping to be a useful tool to validate real-life loans (via statement upload), including missed payment allowances, topups, etc.
    It will take a while to get near to real-life useful, so a POC or a proof-of-motivation to begin with!
    About scenarios, during decision making we should be able to play with "what ifs" and consider impacts, e.g. "what if i will want to topup for a refurbishment in 5 years"
    An important remark in the meantime will be that users should not even be using my model: this should be bank/counterparty-provided! But we are on a journey...
    I can remark about democratisation/transparency, QR codes, web extension...

    Naturally I and users have limited ability to capture/populate product specifics reliably or efficiently, like refinance fees etc. But 1 possibility is to give users a template email/spreadsheet for their bank to populate.

 
 - a household budget model, which can include cashflows from tool above (this demos "unity")
 
 - I can do a similar structured demo of Irish income tax -> household budget model, versus above, but less applicable 
 
 - maybe explorables/explainers/storytelling about finance or loan interest, linking in with above models (not my specific forte)

 Education stuff about maths or simulations like projectiles: simple examples here will always be worthwhile.
 

f) status - development experience

I'm getting models out but admittedly **with a lot of effort**. I need a few fundamentals, I can see that these help everyone:

 - "live/hot editor" with:
 - table of numbers viz or grid
 
Those are the minimum. Also:

 - relationships between numbers (hard without fondue or instrumentation, issue #40)
 - better error reporting (this is hard)
 
A "live/hot editor" is started in github.com/declann/editor based on vega-editor.
"hot data" and "hot version" is worth talking about. (+ -impact)

Here is how I see it working:
 something like @calculang/calculang-hot-editor will be a devDependency in projects
 It will add a command to the project, so that 'npx calculang-hot-editor <calcuconfig.json>' will kick off the magic:
  //placeholder: Currently, that is: compile/run scripts, and launch vega-editor with HMR setup on some datasets

+ Useful visuals: model explorer

g) status - "democratisation"

In the context of this, we need to talk about > developer experience.

Friendly UI is an eventual goal.

Near-term goals:

- Chrome extension experience: into developer tools ((f) above), but also a spreadsheet download purely for understanding numbers (not development)
- calcugithub.com (+calcubitbucket.org, calcugitlab.com for plantform-neutrality): experience like flatgithub.com

+ The mortgage validator example in (e) is important.

y) IDL, ObservableHQ, community

+

community should soon be #1 priority for my public effort.
But ordering is hard: some fundamental changes are already overdue, => hard to break away from pretty patient programmers!

z) misc

near releases: saas example

"Data" viz versus model viz: should we talk about a pivot in this discourse?

Incremental migration <-> process orchestration <-> declarative working
  Future posts/experiments with this.

d3/data mapping <-> input domains/faceting
apparatus is interesting, spread=facet?
My preference so far has been using vega/editor, but for user friendliness need something else.

UI not my main concern because it can also be a separate project, but is interesting.
I'm mostly concerned about this for developers now (hence why vega works)


Probably not for discussion now:

other important stuff to put out:
 - impact models
 - hierarchy stuff and "