Proposal for crystal-lang extension methods (lexical imports and UFCS)

Table of Contents

Rationale for this feature
TL;DR Version
- 'Using' in a nutshell
- "Extension methods" / UFCS in a nutshell
The longer version (nuances and thoughts)

Rationale for this feature

With this document I propose two new features that work together to provide lexically scoped "extension methods" to crystal objects, with the intent of providing a safer, cleaner way to both create and use libraries.

I define "extension methods" as methods that don’t (typically) modify the state of the object they are attached to, nor really make use of OO. Some good examples of existing stdlib methods that should be extension methods include Int.times, the ‘`.to_{x}`’ methods, Enumerable.map and .each.. Basically, anything that would work as a function in a different language.

My reason for desiring a way to safely provide these methods (ie: no monkey patching), is so that libraries can provide (and use) richer, cleaner APIs without fear of stomping on other libs and breaking end-user’s code. Examples of some possible extensions are things like list operations (zipwith, cross), type conversion helpers (.to(AnotherType)), and quality-of-life methods (transform_keys, select_random, rspec’s old, better looking should method)

I honestly want Crystal to flourish, and for this I believe third party libraries need to be able to offer the same rich features and functionality that the standard library provides.

I made a short and long version of the same proposal. Pick which one you want to read.

TL;DR Version

'Using' in a nutshell

Import a group of functions (Module methods) into lexical scope using a keyword (I propose using or import)
- Importing either individual methods, or entire Modules is possible
- Imports can be renamed: using Asdf.bad_name as better_name
- Imports can be unimported: ignore Asdf.better_name (unuse?)
Imports are block/AST-tree scoped
Subsequent imports of the same name overwrite/combine with previous ones
- Allows multiple modules to provide helpers with the same name, for use on different types
- That is why you unimport Asdf.better_name and not better_name in the above example
- No previous_def business.
Libraries can use a macro hook to import defaults into their caller, when loaded with require.
- A second arg to require to suppress this behavior seems like a fair way to control this.

That’s it.

"Extension methods" / UFCS in a nutshell

All method calls that fail to resolve try to fall back to lexically imported methods
- No fancy stuff, just static method calls. So there’s still a rare need for monkey-patches
Imported methods will receive the invocant as their first argument, so multi-method disambiguation applies
Probably resolved before method_missing, as imports are flexible enough to solve clashes

That’s it.

The longer version (nuances and thoughts)

Using Imports

First, I propose a simple "import" mechanism (good keyword: using) that imports Module methods into the current lexical scope ^[1], and this scope is completely separate to any other kind of existing Path-based scope. Module definitions added to this scope shall be given lower priority than any other Path lookup.

Lexical scope will be defined as: "how it looks in the source". All blocks will introduce a new "level" of lexical scope. In theory this should map directly to the AST (before any transformations)

Files should also count as a scope level, and as such no imports can persist beyond the end of a source file.

With those rules given, here is an example of the feature in action:

module Combobulator
  def self.combobulate(list) ; end
end

class SomeClass
  using Combobulator.combobulate

  def entrypoint
    some_array = [1,2,5]
    combobulate(array) # Successful lookup
  end
end

# Combobulate is now out of scope!
class SomeClass
  # `Combobulator.combobulate` is -still- out of scope, even though we are in the same class.

  def combobulate
    # ...
  end

  def run_combobulation(arg)
    using Combobulator.combobulate

    # Imports are lower priority than methods. This should run SomeClass.combobulate
    combobulate(arg)
  end
end

def a_func
  using Combobulator.combobulate

  # As the imports are likely implemented as a stack, adding "unimport" semantics makes sense
  a_proc = ->() do
    # (Could be unuse/unload/remove/hide .. the syntax is unimportant)
    ignore Combobulator.combobulate

    # Can no longer combobulate :<
  end

  another_proc = ->(arg) do
    # Can combobulate :D
    # This lookup should be resolved statically, so works if passed to a callback or something.
    combobulate(arg)
  end
end

Imports should provide fine-grained permissions to work-around name clashes. This syntax is a draft and keywords can absolutely be changed. I have only tried to keep it obvious, simple to parse, and clean:

# Import all methods:
using Combobulator
# Use unimport to blacklist a single method:
ignore Combobulator.recombobulate

# Renaming is important.
using Discombobulator.discombobulate as disc

# Multiple specific imports with renames would be desirable:
using Discombobulator
         .decombobulate,
         .uncombobulate as unco,
         .incombobulate

# `ignore` should also work with lists, as above.

In the event of a name & call signature clash, imports should just shadow any previous matching definitions. Overwriting imports should be considered normal as it allows extending / complementing / replacing other helper methods.

Importing a method of the same name should not overwrite mismatched call signatures. This should allow for some very nice, uniform helper libraries:

using MathMods.double    # Takes an Int32
using StringFixes.double # Takes a String

double(12)     # => Calls MathMods.double(Int32) as it's not matched by StringFixes.double
double("This") # => Calls StringFixes.double(String) as it's not matched by MathMods.double

ignore StringFixes.double
double(12)     # Still fine

One very legitimate flaw in this feature that people know from ruby refinements, is that people are reluctant to have to 'use' a bunch of stuff when monkey patches are more convenient. In order to combat the problem, libraries need to be able to provide 'default imports' when required. This should be possible using a required macro hook that runs in the scope of the importer. A second argument to require disabling this behavior would be a clean and simple way to provide the user a way to prevent this hook from being installed.

Extension methods: Uniform Function Call Syntax

UFCS in crystal should be implemented by falling back to the lexical scope lookup when a method is otherwise not found, passing the invocant as the first parameter. No other rules apply.

One concern is method_missing, which is essentially a similar mechanism. I feel method_missing does not provide the users with enough control compared to an import mechanism so my instinct is to give it higher priority when resolving methods. Maybe adding more control/functionality to method_missing could make the features more harmonious, but that is way out of scope for this proposal.

Here is an example of UFCS in action:

module SuperMaps
  extend self
  # Multi-type dispatch means non-enumerables can be patched to act as enumerables, etc
  def fatmap(mappable : Array | CustomList)
    ...
  end

  # A bit ambitious, but.. untyped helpers could work:
  def thinmap(mappable)
    ...
  end

  def printme(thing)
    puts thing
  end
end

# This is what users see:
using SuperMaps

[4, 3, 2].fatmap &.printme        # Clean! Looks like Crystal code, in fact.

# Would otherwise need to be:
SuperMaps.fatmap([4, 3, 2]) { |a| SuperMaps.printme(a) }

I would additionally propose that the standard lib should move to using extension methods, and then import them into every file’s preamble. This would allow them to be cleanly overridden/enhanced. Some examples would be .times, .hours, .to, and even .map. Eg:

# A domain specific set of tools, such as a numeracy lib:
module MathyThings
  def self.times(a : Int32, b : Int32)
    a * b
  end
end

# A fancier-than-standard Time library:
module SuperGoodTime::Converter
  def self.hours(hours : Int32)
    SuperGoodTime.new(hours)
  end
end

# Speaking of conversion helpers..
# How about a standard method like `into(Type)` for everyone to provide conversions:
module BinaryTree::UniversalConverter
  def self.to(obj : Bytes.class, t : BinaryTree.class)
    BinaryTree.build_from_bytes(obj)
  end
end
# Now able to call 'my_bytes.to(BinaryTree)'.
# It opens up a whole world of inter-library cooperation and easier metaprogramming
# if there's a standard method that everyone can extend.

In closing

While this feature is extremely simple at the core, and therefore might not solve everyone’s problems, it’s important to recognize that this functionality can help devs make very user-friendly library APIs that enhance crystal’s wonderful method-oriented syntax. The addition of simple import controls means end-users get the final say over how things work in their code. And overall, this means a much happier ecosystem :)

Alternatives

Worth listing some other approaches I considered:

Kotlin extension methods: def TargetObject.extraFunction() ; end

[-] Won’t work as plain module methods.
[-] How would renames work? Conflict resolution is important.
[-] Can’t make use of multi-method syntax, have to alias/declare individual types?
[.] "Drop in" functionality, similar to this proposal.
[+] The code body can refer to 'self' as if it was an instance method.
- [-] People might think they can access @vars directly. They’d need to do.. self.@vars. Hmmm.

Ruby refinements:

[-] No renaming. Doesn’t allow the user to resolve conflicts beyond 'not using' a refinement Module.
[-] Super freaking hard to implement? At least if you want to use the next feature:
[+] It can overwrite existing instance methods. Even super works. Actually scoped monkey-patching.
[+] Forward-compatible; Won’t break even if an object adds a method that overwrites an import (could be mitigated with a checker?)

Scala implicits:

[.] Well it’s kinda the same thing as this proposal, I guess.
[-] A little more confusing due to the extra concepts and syntax.

1. It could be current-module scope if people think this is acceptable

Timbus/dontreadme.adoc

Select an option

No results found