Thanks to @demersus, it was discovered that the issue has to do with some irregularity around imports/exports. The solution that was found is to register the format with Quill in the same file it is defined in. Additionally the format must be imported into the snow-with-payment theme to ensure the format is registered.
After struggling for a few days to upgrade the link feature of Quill I've decided to write down my thoughts. To get help from other devs and keep a record of how to do this in the future.
The first step in learning how to add features to Quill is to understand Parchment, Quills internal representation for the DOM. Quill represents elements with its Blots while smaller changes to those elements (like setting an attribute) can be done with Attributors.
For our purposes either would be fine however the primary approach I've taken is to make a 1 to 1 replacement for the link format. Formats are essentially a higher level implementation of blots. Building off the core Blots (block, embed, inline and text) Formats are extended to provide some of the more functional elements like image, link, table, as well as text decorations like bold and italic. In theory we could just create an exact clone of the link format and add an attribute as the link format already does, but in practice its proven more difficult.
In order to have full control over our imports and configuration we've dropped ember-quill and reimplemented it ourselves in art19-ember-app (it also fixes an issue where the actions were losing scope). Currently we extend that component again with lib/art19-core/addon/components/rich-text-editor/component.js, to add basic functionality like grabbing focus or checking to see if the value is empty. I think quill-editor and rich-text-editor could be condensed into one but thats not whats causing the issue so I'll clean it up later. Next we have rich-text-editor/snow-with-payment-link.js which will extend rich-text-editor/component to use the new theme, and will also register our custom Format with Quill.
For our most basic use case we will assume the user is in edit mode, has text in the RTE and has a few characters highlighted. The user will click the link icon in the toolbar which Quill's theme supplies a handler for. Our workflow uses a similar handler which only differs by the choice of Format. On the line in the previous link instructs the SnowTooltip class to put itself in an edit state with a mode of "payment" (our format). From here not much happens until the user clicks the save button which trigger the tooltips save() method.
Finally we see a light at the end of the tunnle.
This save() method has two paths, both lead to calling format() on our value with our new format. With the assurance that our new format will be used we click the save button expecting victory in a tiny blue box. Instead we're greeted with a well fed stack trace.
After following the stack trace for a little, through format() in core/quill.js and into formatText in core/editor.js, you'll be redirrected again to this.scroll.formatAt(), you'll know you went too far if you reach the Delta. "What is scroll?" you might ask. scroll is the confusing way Quill refers to its instance of Parchment. Get it, scroll of Parchment?
Stumbling through the stack trace and into the TypeScript of parchment the issue starts to come into focus. The roadblock in insertBefore is the instanceof check. Now by swapping our editor styles out and using some breakpoints in the script we can see how Quill's own link Format is able to pass this check while ours is not.
If my understanding is correct, by extending from Parchment's Inline blot, our payment format should be an instance of its parent format Inline but it is not, (at least according to instanceof).
Perhapse my understanding of instanceof is incorrect, or maybe I missed a step along the way. If you have any thoughts please reach out.