Skip to content

Instantly share code, notes, and snippets.

@uucidl
Last active October 18, 2022 09:14
Show Gist options
  • Save uucidl/495e7f1c2646fc8b5196 to your computer and use it in GitHub Desktop.
Save uucidl/495e7f1c2646fc8b5196 to your computer and use it in GitHub Desktop.
Notes about APIs

On the emergence of interfaces

Interfaces naturally emerge as software gets broken down into parts communicating with one another. The larger and more deliberate structures emerge from a deliberate attempt to organize the development process itself. [fn:Liskov2008] Structure often emerge directly from division of labor: as teams take on independent tasks, interfaces are established betweeen domains they become responsible for. (Conway’s Law)

Software developers are responsible for systems built out of very small atoms while ultimately performing tasks for their users of a much greater magnitude. Dijkstra showed this by computing the ratio between grains of time at the lowest and largest atoms of the system (from say, CPU instructions to a human interaction with the system) The span was already quite large by Dijkstra’s time, of about 10^9. Today this ratio would be at least above 10^12 (see grain ratios)

This large span has to be managed somehow, often through hierarchies of layers. [fn:EWD361]

[fn:Liskov2008] https://youtu.be/O6By99JW_V8?t=1647 On the desire to partition large systems into modules with intentional interfaces in order to manage the complexity of the software development at scale during the creation of the Venus operating system.

[fn:EWD361] https://www.cs.utexas.edu/~EWD/transcriptions/EWD03xx/EWD361.html

For besides the need of precision and explicitness, the programmer is faced with a problem of size that seems unique to the programmer profession. When dealing with “mastered complexity”, the idea of a hierarchy seems to be a key concept. But the notion of a hierarchy implies that what at one level is regarded as an unanalyzed unit, is regarded as a composite object at the next lower lever of greater detail, for which the appropriate grain (say, of time or space) is an order of magnitude smaller than the corresponding grain appropriate at the next higher level. As a result the number of levels that can meaningfully be distinguished in a hierarchical composition is kind of proportional to the logarithm of the ratio between the largest and the smallest grain. In programming, where the total computation may take an hour, while the smallest time grain is in the order of a microsecond, we have an environment in which this ratio can easily exceed 10^9 and I know of no other environment in which a single technology has to encompass so wide a span.

Inputs

I am looking at practicionners not theoretical material.

Infix vs Prefix

https://simblob.blogspot.com/2019/10/verb-noun-vs-noun-verb.html?m=1

noun verb :: tends to discriminate more than, verb noun. This is interesting in that you want powerful verbs that apply to many nouns, however when you have a noun, you want to see a short list of verbs that apply to it. From specific to generic is better for useability than from generic to specific. See autocomplete.

Don Williamson “Anonymizing APIs” antipattern

> I really, really don’t like “anonymizing” APIs. They take your context-specific knowledge, throw it away, pack your tiny work item into a big generic pipe, then unpack on the other side with code that has to handle the worst case for each work item.

Example: a canvas drawing API that does not differentiate between transparent and non transparent fills means that the renderer implementer now has to differentiate between the two, because the two cases have very different performance profiles. Or something that turns a specific shape into a generic polygon.

Randy Gaul APIs and Game Engines

http://www.randygaul.net/wp-content/uploads/2018/02/R.Gaul_APIs_ITCarlow.pdf

Code Podcast: Don’t Make Me Write UI

https://soundcloud.com/podcastcode/6-dont-make-me-write-ui

Parnas

https://vimeo.com/10556923 at around 01:02 talking about interfaces

Vurtun’s notes about API

API.md

IO Completion Port API

Cited as a good API by Casey Muratori

Akkartik

http://akkartik.name/about

Cautionary tale about making interfaces internal to a system too rigid, which results in them not being improved. It is an angle where the context matters a lot. What is the audience of a piece and its interface? What communication delays are there?

On one end we have systems like the Linux kernel, which try to preserve external as well as internal interfaces stable. On the other end we have software whose internal structures shouldn’t be solidified too early.

Especially since some parts of the break-down that generates interfaces is largely accidental and a result of “problem solving by divide and conquer”

Scott Meyers, things that matter

https://www.youtube.com/watch?v=RT46MpK39rQ&feature=youtu.be&t=27m51s

SCR Navy initiative

http://www.nrl.navy.mil/itd/chacs/sites/www.nrl.navy.mil.itd.chacs/files/pdfs/Heitmeyer2002.pdf http://web.stanford.edu/class/cs99r/readings/parnas1.pdf

Ease of Use without Loss of Power

http://www.pcg-random.org/posts/ease-of-use-without-loss-of-power.html http://www.open-std.org/jtc1/sc22/wg21/docs/papers/2016/p0347r0.html

What If I Don’t Actually Like My Users?

http://ozlabs.org/~rusty/index.cgi/tech/2008-04-01.html

Antirez, Programmers are not different, they need simple UIs

http://antirez.com/news/107

Anteru “Designing C APIs in 2016”

https://anteru.net/2016/05/01/3249/

Jasmin Blanchette “The Little Manual Of API Design”

http://www4.in.tum.de/~blanchet/api-design.pdf

Michael Bostock (d3)

http://twitter.com/mbostock/status/681561150127878144

It’s funny how writing documentation can spur redesign: it’s easier to simplify a complex API than try to document it completely”

What Makes Software Good

Rule of thumb by Josh Bloch

If you can’t come up with a good name for a method or a class, your design might be flawed

CBLOOM

09-19-15 | Library Writing Realizations http://www.cbloom.com/rants.html

Stepanov’ lectures

Hard to use those from the Efficient Programming with Components

Efficient Programming With Components

Lecture 7 part 2

At that time Stepanov reveals that actual usage is necessary to write the generic operation:

https://youtu.be/S2iTfUyVOcY?list=PLHxtyCq_WDLXryyw91lahwdtpZsmo4BGD&t=273

Most of the times writing a program is an exploratory walk.

Lecture 11 Part 1, where he briefly mentions iterators and boost ranges:

https://youtu.be/84gHZgPCf1s?list=PLHxtyCq_WDLXryyw91lahwdtpZsmo4BGD&t=1455

Good choice of operator https://youtu.be/2mU8CTO2vSc?list=PLHxtyCq_WDLXryyw91lahwdtpZsmo4BGD&t=2922

And the reasoning behind advance and distance vs +=

https://youtu.be/_8hN232WNYU?list=PLHxtyCq_WDLXryyw91lahwdtpZsmo4BGD&t=276

You cannot know if you’re right until you found thing in context.

https://youtu.be/Dly8Ff4aDp8?list=PLHxtyCq_WDLXryyw91lahwdtpZsmo4BGD&t=1797

Write usage code first

The interface of rotate is designed to make it easy to compose. Its return value.

Other principle: don’t throw away information that your algorithm has already computed.

Per vognsen + Vurtun

https://gist.github.com/pervognsen/d57cdc165e79a21637fe5a721375afba https://gist.github.com/vurtun/192cac1f1818417d7b4067d60e4fe921

My notes on algorithm lib

  • Write Usage Code First
  • Don’t Throw Away Information that was computed
  • Iterate

Write Usage Code First and Iterate

An algorithm should first be written in context i.e. the usage code is written first and the algorithm extracted from it.

It is even better if you have experimented with multiple usages so as to discover the best interface for the algorithm. Don’t worry however and be ready to revisit your API multiple times, as it is impossible to get right on the first try.

Don’t Throw Away Information

Anything that your algorithm is computing as part of its operation is of potential use by the user of this algorithm: don’t throw it away!

An example is a find function. It could return whether it found an element or not (bool) However in the process of finding that element it must have known the position of that element. So better return the position and not only a boolean. The position will be of great use for the user of your algorithm.

Simon Cozens on implementing subsystems

This is more about exploratory design than anything else.

Simon Cozens in a FOSDEM2015 talk about the SILE typesetting system:

https://youtu.be/5BIP_N9qQm4?t=1282

When you are implementign a subsystem, always implement it more than once. (…) that tells you where the separation of concerns lies.

One of the best piece of programming advice I ever got was from a guy I worked called Tony Bogen. He said that when implementing a subsystem always implement it more than once. And have two or three different variations of that because that tells you where the separation of concerns lies. And that’s been very helpful. Everytime I don’t do this I come to regret this because I generally end up writing that subsystem multiple times anyway.

Towards a Theory of Software Design by Daniel Jackson

https://youtu.be/cNe6g0qczxE

Of particular interest its categorization of bad aspects, bad applications of concepts at the design stage, and its critique of existing software designs, including git.

See my “Commentary And Notes” notebook.

Reference: Granny Aninmation System 2.0

A common goal is to achieve code reuse.

Introduces the notion of Integration Discontinuity. This is basically what happens when the relationship between the user and the component being used starts growing to a point where the reused component does not provide what’s necessary for its user.

Theory and practice?

A component promises something but how is it in practice. Granny, first version built on reasonable code principles. Lots of people had problems integrating it. What happened?

The second incarnation was built after we learnt about it and this is our assesment

There are multiple tiers as you start integrating an API. They correspond to multiple stages of integration.

These tiers call for various characteristics. The idea though is that you let users of your API move between phases of integration so that they keep options open and don’t suffer from a discontinuity as they encounter a block.

https://vimeo.com/157022266 min 58:00 w/ Fabian Giesen

Qt notes about API

http://wiki.qt.io/API-Design-Principles

The Boolean Trap

https://gist.github.com/uucidl/68d471b05c3a82d0f0556274f57cf6a3 http://ariya.ofilabs.com/2011/08/hall-of-api-shame-boolean-trap.html

tl;dr:

A good API is optimized for reading code not writing. One easy to spot API mistake is the “Boolean Trap.” It can be summarized with the following rule: “It is almost invariably a mistake to add a bool parameter to an existing function.

Also https://blog.ometer.com/2011/01/20/boolean-parameters-are-wrong/

TODO: enumerate the effects of adding a bool to an API entry point, depending on various properties of that entry point and the generated effect. (With respect to granularity/orthogonality/stability of interface etc..)

I’m now super careful with adding bool parameters to the public imgui api. Many bools became an api mess/issue in the long run.

So I have a public function and want to add a bool parameter to it. Other solutions seem overkill for now (Two entry points? Flags? Enum?)

Joshua Bloch’s “How To Design A Good API and Why it Matters”

https://www.youtube.com/watch?v=heh4OeB9A-c

Arvid Gerstmann

@url: https://twitter.com/ArvidGerstmann/status/918757328920473600

<Avid Gerstmann> I said it before and I’ll say it again: ZeroMQ is the gold standard of API design. Super simple C API, yet incredibly powerful: http://api.zeromq.org/ The guide is a good read, although quite long: http://zguide.zeromq.org/page:all

Yes, it’s C, but it’s easy to built your own C++’ey abstraction on top of it, like you see fit. Many C++ APIs are too opinionated, or use STL. ZMQs C API is low-level & flexible. I dislike having a high-level API & prefer building it myself. ZeroMQ also gets this right. They offer CZMQ, a separate high-level library, wrapping the low-level library & providing a nice API.

Butler W. Lampson Software Components, Only the Giant Survive

http://research.microsoft.com/en-us/um/people/blampson/70-SoftwareComponents/70-SoftwareComponents.htm an answer by stepanov: http://www.stepanovpapers.com/Industrializing%20Software%20Development.ppt https://softwareengineering.stackexchange.com/questions/221615/why-do-dynamic-languages-make-it-more-difficult-to-maintain-large-codebases/221658#221658

on that topic: http://www.uh.edu/engines/epi1252.htm http://st.inf.tu-dresden.de/files/teaching/ss10/cbse/01-introduction.pdf http://www.cl.cam.ac.uk/~srk31/research/talks/kell16operating-slides.pdf

Butler W. Lampson, Hints for Computer System Design

https://www.microsoft.com/en-us/research/publication/hints-for-computer-system-design/ https://www.microsoft.com/en-us/research/wp-content/uploads/2016/02/acrobat-17.pdf

Parnas D.L. On The Criteria To Be Used in Decomposing Systems Into Modules, Comm. acm 15 12, dec 1972 p 1053-1058

http://repository.cmu.edu/cgi/viewcontent.cgi?article=2979&context=compsci https://www.cs.umd.edu/class/spring2003/cmsc838p/Design/criteria.pdf

Britton, K..H, et al. A procedure for designing abstract interfaces for device interface modules. Proc. 5th Int’l Conf. Software Engineering, ieee Computer Society order no. 332, 1981, pp 195-204.

On compatibility layers

“The people who rely on the compat layers don’t care enough to maintain it. The people who work on the mainline system don’t care about the compat layers because they don’t use them. The cultures aren’t aligned in the same direction. Compat layers rot very quickly . ” – Theo De Raadt

Roy Fielding Architectural Styles and the Design of Network-based Software Architectures

aka the REST API paper

Objects Have Failed by Richard P. Gabriel

“The strong typing of object-oriented languages encourages narrowly defined packages that are hard to reuse. Each package requires objects of a specific type; if two packages are to work together, conversion code must be written to translate between the types required by the packages.” [ John K. Ousterhout]

Also notes about reuse’s failure.

Monocipher write-up

http://loup-vaillant.fr/articles/implemented-my-own-crypto

API Design Tips For Libraries by André Staltz

http://staltz.com/api-design-tips-for-libraries.html

Frameworks are fundamentally broken by Timothy Perrett

http://timperrett.com/2016/11/12/frameworks-are-fundimentally-broken/

Example of the value of naming conventions

https://www.sebastiansylvan.com/post/matrix_naming_convention/

Antirez on Rax vs Dict

Reading https://twitter.com/antirez/status/958028135605383169

See Rax api at: https://github.com/antirez/rax

After delivering a shitty API like Redis’s dict.c hash table, this time I did my homework well and the rax.c (radix tree) API is really pleasant to use.

Dict API:

#if 0
// Macros:
dictFreeVal(d, entry)
dictSetVal(d, entry, _val_)
dictSetSignedIntegerVal(entry, _val_)
dictSetUnsignedIntegerVal(entry, _val_)
dictSetDoubleVal(entry, _val_)
dictFreeKey(d, entry)
dictSetKey(d, entry, _key_) 
dictCompareKeys(d, key1, key2)
dictHashKey(d, key)
dictGetKey(he)
dictGetVal(he)
dictGetSignedIntegerVal(he)
dictGetUnsignedIntegerVal(he)
dictGetDoubleVal(he)
dictSlots(d)
dictSize(d)
bool dictIsRehashing(d)
#endif

dict *dictCreate(dictType *type, void *privDataPtr);
int dictExpand(dict *d, unsigned long size);
int dictAdd(dict *d, void *key, void *val);
dictEntry *dictAddRaw(dict *d, void *key, dictEntry **existing);
dictEntry *dictAddOrFind(dict *d, void *key);
int dictReplace(dict *d, void *key, void *val);
int dictDelete(dict *d, const void *key);
dictEntry *dictUnlink(dict *ht, const void *key);
void dictFreeUnlinkedEntry(dict *d, dictEntry *he);
void dictRelease(dict *d);
dictEntry * dictFind(dict *d, const void *key);
void *dictFetchValue(dict *d, const void *key);
int dictResize(dict *d);
dictIterator *dictGetIterator(dict *d);
dictIterator *dictGetSafeIterator(dict *d);
dictEntry *dictNext(dictIterator *iter);
void dictReleaseIterator(dictIterator *iter);
dictEntry *dictGetRandomKey(dict *d);
unsigned int dictGetSomeKeys(dict *d, dictEntry **des, unsigned int count);
void dictGetStats(char *buf, size_t bufsize, dict *d);
uint64_t dictGenHashFunction(const void *key, int len);
uint64_t dictGenCaseHashFunction(const unsigned char *buf, int len);
void dictEmpty(dict *d, void(callback)(void*));
void dictEnableResize(void);
void dictDisableResize(void);
int dictRehash(dict *d, int n);
int dictRehashMilliseconds(dict *d, int ms);
void dictSetHashFunctionSeed(uint8_t *seed);
uint8_t *dictGetHashFunctionSeed(void);
unsigned long dictScan(dict *d, unsigned long v, dictScanFunction *fn, dictScanBucketFunction *bucketfn, void *privdata);
uint64_t dictGetHash(dict *d, const void *key);
dictEntry **dictFindEntryRefByPtrAndHash(dict *d, const void *oldptr, uint64_t hash);

Rax API

/* A special pointer returned for not found items. */
extern void *raxNotFound;

rax *raxNew(void);
int raxInsert(rax *rax, unsigned char *s, size_t len, void *data, void **old);
int raxRemove(rax *rax, unsigned char *s, size_t len, void **old);
void *raxFind(rax *rax, unsigned char *s, size_t len);
void raxFree(rax *rax);
void raxFreeWithCallback(rax *rax, void (*free_callback)(void*));
void raxStart(raxIterator *it, rax *rt);
int raxSeek(raxIterator *it, const char *op, unsigned char *ele, size_t len);
int raxNext(raxIterator *it);
int raxPrev(raxIterator *it);
int raxRandomWalk(raxIterator *it, size_t steps);
int raxCompare(raxIterator *iter, const char *op, unsigned char *key, size_t key_len);
void raxStop(raxIterator *it);
int raxEOF(raxIterator *it);
void raxShow(rax *rax);
uint64_t raxSize(rax *rax);

One aspect that makes it good, is that we can see this API apply directly to a hash table like dict almost out of the box. As long as the user is willing to convert their key types to bytes. Some may consider writing these bytes as too expensive, however I’d argue it’s easier to get bad performance out of injecting equality and hashing callbacks into a hash table. The hashtable has to be able to compare for equality, afterall.

Flutter Design Principles

@url: https://flutter.io/design-principles/#introduction @title: Flutter Design Principles

Breaking API changes are treated as specific items worthy of attention. Proposals are written and summarized.

  • Justification is made
  • A migration path from old to new code is described
  • Contact for supporting people to move old to new API

Weighting of API stability versus benefits will determine whether the breaking change is to be made.

Transition period by way of annotating old code as deprecated is to be introduced. (using tags such as @escape{@deprecated(‘Description’)})

  1. Avoid hard to maintain data-retention/duplication

@quote{There should be no objects that represent live state that reflects some other state, since they are expensive to maintain. e.g. no HTMLCollection}

I.e. they discourage data-retention, as it requires synchronization between layers. HTMLCollection

  1. Easy access implies cheap access

If something is implemented via what looks syntactically cheap, it should also be cheap.

  1. Expensive operations should not be exposed as synchronous procedure calls
  2. APIs should be arranged in “physical” levels.
  • Convenience APIs layered on top of lower-level APIs
  • Scope of each level is made as narrow as possible
  1. Unsafe constructs are not promoted as regular APIs

ex: low-level construction of executable code from unsafe/user input pieces

  1. Adapter APIs should be complete

When wrapping another API, faithfully wrap the complete API so as to minimize surprises and avoid creating integration surprises.

Kiwi Guide To APIs

@url: https://code.kiwi.com/code-design-principles-for-public-apis-of-modules-6a43aaf26624 @title: Code Design Principles for Public APIs of Modules

Main actions on a module:

  • Creation from scratch
  • Redesign

Heuristics

Names

Remove anything that appears unnecessary (information compression) .. without going as far as hurting redability.

Example of a name that can be shortened, kw.booking.additional_booking_management could be better written kw.booking.additional_booking, as “management” does not appear to contribute much meaning.

Goals for better names:

  • legible useage code, emphasizing its own logic (names that are too long might disrupt legibility of the useage code)
  • names that are easy to be kept consistent (i.e. don’t invite variations)

Insights for better APIs

  • how many engineers would maintain that module
  • how many engineers would commonly write code using that module
  • how frequently code using this API might be read over time (times/day)
  • how long might this module be used
  • consider what the user will see

This should help you decide the level of quality this API and implementation should be at.

Before making an implementation clean, think about the user first, and make the API clean. Write tests, prototype useage code.

Procedures

  • keep number of positional parameters small (< 3)
  • procedures are designed to work on data (if the procedure of a verb, its object), so design/specify that

Hashing API

<@pkhuong> Oct 9 Reading Fluent Python… does the official Python documentation really recommend hashing objects by combining their fields’ hashes with xor?

<@pervognsen> Oct 9 Huh? I’ve always just hashed a tuple of the fields…

I’ve definitely seen recommendations like that in various Java books, sadly, but Python makes it so easy to just hash a tuple.

Incidentally, one of my favorite general tricks is to exploit tuple isomorphisms for boilerplate code like that.

<@pkhuong> Oct 9 For sure. Much easier than writing our all 6 comparator methods by hand (:

I assume you also hash in the class name :p

I’ve always thought that hash methods should return an opaque hash_t type, so you aren’t even tempted to do this kind of crap.

<@pervognsen> Oct 9 I think the right interface for extensible hashing is closer to serialising data to a stream (that happens to hash its input bytes). I saw a C++ standard proposal like that; I don’t know what happened to it.

<@pervognsen> Oct 9 Yeah, I was about to correct myself. You need that for streaming. Then you just provide variadic helpers for the combiners. And it lets you distinguish internal hash state vs final hash value.

I prefer the “fold/append/mix into hash state” API for a systems programming language since it doesn’t involve other intermediate objects.

Hrm, I vaguely remember the C++ proposal you’re referring to. I don’t think it was ratified, but I recall it used the streaming style.

<@matt_dz> Oct 9 A couple, most recent in 2015 (http://wg21.link/P0029 ) and 2016 (PDF: http://wg21.link/P0199 ); latter withdrawn: https://botondballo.wordpress.com/2016/07/06/trip-report-c-standards-meeting-in-oulu-june-2016/ ….

On Drag And Drop APIs And Their Generality

@url: https://twitter.com/ocornut/status/941627778910380032

<Omar Cornut> dear imgui: figuring out how to interface with Windows drag and drop, and it turns out that when using WM_DROPFILES it is trivial but super limited, and anything else seems like pain.

<Leonard Ritter> the windows drag & drop API seems really weird until you implement your own and discover all these corner cases and go “oh. okay. now i get it”. the GTK one is just as crazy. For my own imgui-based editor I built a drag & drop system that was purely internal, and that alone got fairly complex already. it’s like a network protocol that goes “i have these formats to drop, which one is fine?” – “these are fine” – “can i drop this one?” – “you may” And because, for the desktop context, it’s also IPC, you need mimetypes or some other global enumerator so y’all understand each other.

<Leonard Ritter> In an earlier windows-based UI for an audio app i ended up using the d&d system for all kinds of stuff (including a dsp node editor) because it was so versatile.

<Omar Cornut> What’s annoying is there are so few good examples of how to support Win32 DND in your app, and once you get past WM_DROPFILES you have to do the whole thing. I was wondering if we can just somehow more easily implement a IDropTarget to get a better WM_DROPFILES with preview.

<Omar Cornut> There’s a beta drag and drop api in imgui master if you want to check it out for suggestions (its basically 6 functions). It’s missing demos but there are a few uses of it in the code.

<Omar Cornut> Drag target can compare data format (which are essentially strings in dear imgui land). The main thing that missing now are A) the possibility to transport multiple data formats simultaneously B) the possibility for the source to compute the payload later (e.g. on drop)

Slug (Decoupling Allocations)

https://twitter.com/EricLengyel/status/938590610604175360

<Eric Lengyel> Slug was designed to have no dependencies whatsoever. It doesn’t make calls into anything external to itself, including the standard library, any rendering API, and the operating system. It doesn’t even allocate memory.

<Gustavo Samour> The API gives you the buffer size, you allocate, and then pass in the pointers back to the API

<Eric Lengyel> Same goes for vertex buffers. The CountSlug() function tells you how much space you need to allocate for the BuildSlug() function to fill with vertex and triangle data. (Slug does not use callback functions to perform allocations.)

<Dale Kim> I feel that this style of API needs to have more visibility. A lot of folks see a library as code that does something for you, but many libraries do this by also taking away some of your control. Memory being owned by the library is a common example.

Whether or not the API controls you or you control the API is a central issue I see in many problematic systems. In most cases, you want the caller to retain as much control as possible.

<Tom Forsith> It’s OK to ship with defaults to do those things. Granny is the same - if you do nothing, it uses a bunch of standard OS calls. But you can override them easily, either by callback or pre-emption.

<Bryan McNett> even then, i like my libraries to treat allocations as special events, only to be done predictably and rarely.

<Tom Forsith> Agreed. “Create” should be in the name of the function, etc.

Succesful Interfaces

  • MIDI
  • x86 ISA
  • SQL
  • OpenGL
  • Any programming language

These are examples of well-defined interfaces which have allowed big components reuse and their independent improvement/optimization.

For example, a programming language is a well defined interface, to reusable components (compilers) creating computing automatons (programs)

Ideas

Useability

Useability is how well (speed, effort) an user can achieve a goal, within a certain context.

Ex: “Given that I don’t know what to find, and want to search the internet, google’s search, with its single text entry and quick results gives me an effortless and quick list of potentially interesting web links”

Properties of APIs

BadGood
Opaque dataTransparent dataEfficient Basis
CategoryGoodBad
RedundancyOffers smooth gradiant for integration
RedundancyAccomodates user types
RedundancyNoisy, hard to grasp
Low GranularityFlexibilitySimplicity
CouplingInflexible
CouplingDefects easy to create

Relationship between idempotence and declarativeness. I.e. you can apply f as many times as you want because it encodes something about the end result, not an action to be perform. For pure functions it is easy, for procedures the trade-off is in some form of state retention.

Constructing examples

Tadashi Takieda says:

count(Def) < count(Theorems) < count(Examples)

count(Def) < count(LogicalResults) < count(Examples)

https://www.youtube.com/watch?v=J7vojBbvudQ&feature=youtu.be&t=139

Keywords, bits of inspiration

“Design Of Everyday Things” ; Affordances ; Design of clean programming interfaces: tools coming from Design, tools coming from Mathematics ; Mathematics is mostly about user interface ; Theories and concepts as user-interfaces

Industrial design has traditionally seen itself as a way to attract and seduce a customer (see “Never Leave Well Enough Alone” by Raymond Loewy) Some amount of surprising arrangements, within the constraints of serving logically the function of an object was therefore necessary. It seems mostly useless and slightly harmful to care about this aspect for internal modules. It could be a factor of success for an opensource or commercial library.

In the book see his example of the egg as an ideal form perfectly adapted to its function. Balance between strength and aerodynamics. The principle of economy (of materials for instance) leads to elegance.

Christopher Alexander

A first principle of construction: on no account allow the engineering to dictate the building’s form . . . . never modify the social spaces to conform to the engineering structure of the building. –Christopher Alexander

Tom Forsyth

“Abstraction”

https://twitter.com/tom_forsyth/status/924692979721281537

My rule is - when writing at layer X, if you can’t easily say what the layer below looks like, you’ve abstracted too far.

Usage

Traditional software engineering wisdom says

wisdomcounterpoint
Target an interface, not an implementationHard to do if you don’t have two implementations
Interfaces should hide change-prone detailsPlanning fallacy, risk of hiding interesting problem domain details
Don’t Repeat YourselfCreates central points of failure and bottlenecks

Styles

  • Retained/Immediate (Q. about data-retention / automation)
  • Push/Pull (Q. about latency / single vs multiple control flows)

Evolution over time

Spec-ulation keynotes by Rich Hickey

https://www.youtube.com/watch?v=oyLBGkS5ICk

There are two types of changes to an interface: grow or break. At every level of what an API provides and requires:

Grow: provide more, require less Break: provide less, require more

Levels:

  • artifacts
  • names
  • functions

Problem with middle layers in the Linux kernel

https://lwn.net/Articles/336262/

problem with the supposed to be “secure” extensions to traditional C functions

http://www.open-std.org/jtc1/sc22/wg14/www/docs/n1967.htm

Dave Cutler & Darryl Havens

Darryl Havens (responsible for the I/O system on NT)

“If you spend that amount of time designing something and you have a spec that gives you every single API [application programming interface], what its inputs are, what its outputs are, you pretty much know how the thing is going to work,” Havens said. “So I actually typed in the code for the entire I/O system in three weeks. That’s how well-designed it was. By the time I sat down to write the code, I already pretty much knew how it was going to work.” Read more at https://news.microsoft.com/features/the-engineers-engineer-computer-industry-luminaries-salute-dave-cutlers-five-decade-long-quest-for-quality/#ydU2bC61j8j51kq6.99

https://news.microsoft.com/features/the-engineers-engineer-computer-industry-luminaries-salute-dave-cutlers-five-decade-long-quest-for-quality/#hskvWsfxAHmVR0Qq.97

Fabian Giesen on stb_image

++number_of_people_who_reported_the_documented_stb_image_behavior_as_a_bug; (It’s the “channels_in_file” output parameter. The #1 API regret in all of the stb libraries, but fixing it would break backwards compat, so @nothings and me have shied away from it.)

We’re thinking about a major release at some point that groups a bunch of API changes (including removing some of the more obscure formats, getting rid of the globals for error reporting, and cleaning up the currently messy 8-bit/16-bit/float situation), but this is the #1 thing.

every API designer has some regrets, but it’s darkly funny even stb_image, with a primary API consisting of one entry point, has That One Thing.

Software Engineering, a practictioner approach by Roger Pressman

He names “patterns of control” which can be useful to characterize an API.

In the sequential pattern of control, the caller hands execution fully over to the module. In the incremental pattern of control, the module and caller alternatively take the execution control. In the parallel pattern of control, the work of the module is concurrent with its caller.

Books

https://www.amazon.com/API-Design-C-Martin-Reddy/dp/0123850037

Granularity. git has fine and coarse commands.

I use git gui quite regularly to add/commit/amend messages.

Here is a list of commands I perform regularly:

cmdocc
git-format-patch1
git-patch1
git-reflog2
git-rev-list2
git-tag2
git-blame3
git-init4
git-gc5
git-prune5
git-for-each-ref6
git-bisect9
git-worktree13
git-am17
git-merge18
git-remote20
git-clone21
git-config27
git-stash27
git-revert28
git-mv47
git-rm52
git-show57
git-log89
git-cherry-pick111
git-ls-files117
git-clean126
git-pull152
git-grep240
git-submodule246
git-reset265I often do `git fetch ; git reset –hard` instead of git pull
git-describe301uh? probably my prompt
git-commit319
git-add393
git-fetch432
git-checkout555
git-push574
git-branch618
git-rebase682
git-status902
git-rev-parse17172command prompt

What is the problem domain that git solves for me?

My perspective is that I use git to:

  • keep up to date with most recent version of the software I’m working on,
    • ((git fetch ; git rebase or git reset | git pull) ; [git push –force-with-lease])
  • make local changes safely w/ annotations (why) and rollback,
    • (git add ; git commit -m | git gui)
  • publish changes to automated test systems
    • (git push -u origin <branch-name>)
  • prepare changes for review by colleagues,
    • split changes in atomic digestable micro changes
      • (git rebase -i origin/<base-branch> ; emacs + git gui)
    • annotate changes with messages
      • (git commit | git gui)
  • publish changes for review by colleagues,
    • (git push –force-with-lease)
  • publish changes officially to the codebase (sync to main line, then publish)
    • grab lock for main line
    • keep up to date
    • command line (git merge –no-ff ; git push) | github merge (most often, so I can copy information)
    • <other routine tasks>
    • release main line lock

That problem domain is almost entirely defined by the tool of version control itself, of which many tools exist.

API Patterns: Immediate Mode UI

In the dimensions of API design. Where does the Immediate/Retained axis lives?

DimensionSketchTrade offExamples
Granularity`A` vs `B; C`Flexibility vs simplicity
Redundancy`A` vs `B`Convenience vs OrthogonalityComplex number construction
Coupling`A implies B`Performance, Convenience vs SafetyMemory management
Associativity`(A; B) / C` == `A / (B ; C)`
Retention`e = E(); A(e)`Automation vs Performance/FlexibilityDatabases
Top-down flow controlCounter-example: Frameworks

We are talking about two API styles. Retained and Immediate APIs. Although their canonical examples are the construction of user interfaces, these styles are adopted in various other domains.

Retained API:

  • the user pre-declares entities and their relationships,
  • dynamic properties of the system, behaviors are injected as entities, (callbacks, visitors, listeners, controllers) and associated with the entities,
  • control flow is spread across the network,
  • network of entities must be mutable, to allow to dynamically add a number of entities that are unknown at compile time,
  • user owns reference to entities,
  • entities must be explicitely synchronized with the application’s data model,

If data is stored on two sides of an API boundary then you are forced to do synchronization whenever mismatches exist. It is worth it when the automation that the API implementation can perform as high-value compared to the complexity and cost of synchronization.

Example of natural retention: databases, filesystems.

Immediate API:

  • uses code/function calls to declare and implement effects,
  • composition of entities is created using function composition,
  • data model is translated into effects and the application reacts to changes directly where these effects are declared
  • presumes that the application can know when data has changed, and that it know what to redeclare,
  • implementation usually must be able to diff, cache and clip for performance reasons,
  • top down control flow that’s easy to reason about (no callbacks),

Note that in case you have 60hz animation, detecting that something has changed on the screen is a non-issue: there’s almost always something changing on the screen. So the question is rather what has changed and how to update it quickly.

Typical problems with immediate mode ui is layouting. I.e. global constraints that connect multiple entities.

Context for retained APIs. Different computer architectures? Did Xerox PARC use retained mode APIs? This can actually be checked at [http://xeroxalto.computerhistory.org/xerox_alto_file_system_archive.html] After reading a bit this is mostly procedural code with state w/ top down control flow. The Laurel email client written in Mesa has something called BuildScreenStructures that reminds me of traditional retained APIs. See http://xeroxalto.computerhistory.org/Indigo/DMS/Laurel/6/.InitInteractor.mesa!1.html

The style that immediate API follows can be summed up with the following slogan: “Leave it to the client” (Butler W. Lampson, Ref.001)

We need to demonstrate using an example.

Ideas:

  • C++ native app ALF/Qt/Dear IMGui?
  • HTML App

References

Leave it to the client. As long as it is cheap to pass control back and forth, an interface can combine simplicity, flexibility and high performance by solving only one problem and leaving the rest to the client. For example, many parsers confine themselves to doing context free recognition and call client-supplied “semantic routines” to record the results of the parse. This has obvious advantages over always building a parse tree that the client must traverse to find out what happened.

Example implementations & stuff to read

Practices

Information hiding & modularity

  1. use a module by reading its documentation only,
  2. if a doubt arises and you happen to read the implementation, update the documentation

Other Notes

retained vs immediate (gl/dx/imgui/qt) stateful vs stateless (see EJB) heap vs stack

Immediate UI rely on ID to keep track of an element across frames of operation.

“` \Exists c, cT \Suchas cT=Type(c) ∈ Controls: \Forall d ; id(c) = id(d) \implies c = d “`

Most controls have labels. The label can often conveniently be used as an id. Two counter examples:

“` \Exists d, c \Suchas Type(c) ∈ Controls ^ Type(d) ∈ Controls: \Exists f ∈ Frames: label(d, f) = label(c, f) “`

“` \Exists c \Suchas Type(c) ∈ Controls: \Exists f_0, f_1 \Suchas label(c, f0) != label(c, f1) “`

Retained vs Not-retained API oscillations

As you cross boundary layers there might be a tendency to notice that:

  • from events people create a static representation (snapshot value)
  • from multiple values people generate by difference the events that could be observed (because that corresponds more neatly to the work to be done)

“` A_0 (db1) A_1 (db1) -> add/remove events -> B_1 -(diff w/ B_0)-> add/remove operations -> C_1 “`

IM for physics

http://chipmunk-physics.net/forum/viewtopic.php?t=1434 http://twvideo01.ubm-us.net/o1/vault/gdc09/slides/gdc09_insomniac_physics.pdf https://github.com/erwincoumans/PhysicsEffects

In math, functions are often described as: \(f : Domain ⇒ CoDomain\)

In programming, where we have many names, a short nice name like f isn’t available a longer name is needed.

In C-like languages this translates to something like:

Planet x;
double y = PlanetVolumetricMeanRadiusInE6KM(x);

This emerges from a need to find a trade-off between:

  1. ease of enumeration of all transformations applied to a Domain (e.g. via auto-complete)
  2. ease of identification of the result entities of a transformation (especially for large names)

The former leads us to prefix the function names with their domain to allow auto-completing symbols using the domain type of the function as a starter stem.

In a typical OOP-style we obtain instead:

Planet x;
double y = x.VolumetricMeanRadiusInE6KM();

Here the the result entities are clearly highlighted while the domain is hidden, and auto-complete requires some type-inference or context awareness.

Functions are also analog to maps. What is the map syntax for this?

It is either:

map<Planet, double> VolumetricMeanRadiusInE6KMForPlanet;
auto y = VolumetricMeanRadiusInE6KMForPlanet[planet];

or

map<Planet, Point> PlanetVolumetricMeanRadiusInE6KM;
auto y = PlanetVolumetricMeanRadiusInE6KM[planet];

Trade-offs at play:

  • Friction between map representation vs function
  • Namespace organization w/ auto-complete vs result entities clearly highlighted

However if there’s one thing where maps aren’t analoguous to C/C++ procedures, it is in that a map only has a single “argument.”

The point of low friction is clearly with the pair:

  1. procedure: src_C{y = PlanetVolumetricMeanRadiusInE6KM(x)} and,
  2. map: src_C{y = PlanetVolumetricMeanRadiusInE6KM[y]}

[[https://github.com/uucidl/pre.uumu]]

Implementing Per Vognsen's Mu API

Per Vognsen designed the Mu API as an alternative to libraries like SDL, SML. I find myself agreeing with many of its design decisions.

He documented it at:

His goals:

  • minimal platform layer for multimedia apps (in terms of binary/source size)

  • not industrial-strengh, but excellent for small apps

  • experiment with API design:

    • good, ritual-free, defaults
    • eschews the many granular calls these libraries tend to have to set it up or get information from it
    • instead, provide data in/out through datastructure and a minimal set of functions
    • dialogue between the app and the library as if the library was a resumable coroutine, to minimize callbacks, which disrupt normal code flow
  • extension API with media file loading (image, sound, video)

This is achieved with a global datastructure, whose entries serve as much as possible as both input and output.

Other principles:

Precompute redundant data that is most often useful to users, rather than doing it lazilly and potentially too often. Even if that's increasing the surface area of the API.

Provide state of controls rather than exposing an event queue to its users, since since multimedia applications would anyway have to sample/refresh at a high enough rate anyway.

In cases where naïve state-capture would lose events, such as for text, the API prepares buffers of prepared data representing the aggregate input between two frames.

Compromises:

As of stream #2, audio has been implemented via a callback, pulling samples regularly from the high priority audio thread.

Experiments to try:

  • The input/output struct is plain old data (if you except the platform specific handles) ; which means it should be trivial to record sequence of values to replay back the application eventually.

Some other personal comments:

A push API for audio is certainly possible, however I personally think if it make sense, it should prevent gaps in audio as the result of a late frame. It does simplify the usual, simple cases because it removes the need for thread-safe code.

Another comment I can make is that the choice of int16_t for audio samples, while convenient for mixing in samples coming from audio files, make synthesis cases less natural. It's easier and less error prone to use float within the 0..1 interval as a generic representation of audio samples.

Data-oriented APIs lead to this quite nice low ritual style.

If there's one negative thing about it, it's that we are used to think in terms of API breakage on the basis of an API's functions, however people are not used to detect and document API changes due to data changes. This might be why a lot of APIs chose instead to use opaque data types and lots of functions.

If however one thinks about the larger issue of ABI compatibility then it would be a good idea to invest in tooling to detect data changes anyway, and the difference disappears.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment