Things slime does

thingsslimedoes.md

quick survey of swank interfaces in order to see what could be broken out into proper portability libraries or, more optimistically, language extensions.

this is preliminary. i pretty much just went down swank.lisp looking for definterfaces. Some other portability related things are not covered yet.

Encoding/decoding

Babel

• Encode a Lisp string to an octet vector of its UTF-8 encoding
• Decode an octet vector of UTF-8 to a Lisp string

Sockets, TCP/IP

Maybe the sb-bsd-sockets interface is adequate for this? Several implementations export it, or at least SBCL, ECL, Clasp. Failing that there's usocket.

• Create and close sockets for a host and port (create-socket, close-socket)
• Get the port number for an existing socket (socket-port)
• Accept a connection on a socket (accept-connection)
• Set up asynchronous handlers based on SIGIO or on a file descriptor (add-sigio-handler, remove-sigio-handlers, add-fd-handler, remove-fd-handlers)
• Set the timeout for a stream (stream-timeout)

Unix

• getpid
• Install handler for SIGINT (install-sigint-handler, call-with-user-break-handler)
• Get the name of the executable i.e. argv[0] (lisp-implementation-program)
• Get the FD for a socket stream (socket-fd)
• Make a character stream for a FD (make-fd-stream)
• Duplicate a fd (dup)
• Replace the process with a new executable (exec-image)
• Get argv (command-line-args)

Pathnames

A comment says "pathnames are sooo useless". I think UIOP handles this stuff?

• Interconvert filenames and pathnames without e.g. interpreting wildcards (pathname-to-filename, filename-to-pathname)
• Access default directory, or the working directory I guess (default-directory, set-default-directory)

Package local nicknames

I think we're all basically using SBCL's interface.

• Get the local nicknames of a package as an alist with the home package and the nickname (package-local-nicknames)
• Look up a package by local nickname relative to a home package (find-locally-nicknamed-package)

Compiler (source locations, diagnostics)

• Define hooks to be called when the compiler encounters a problem (call-with-compilation-hooks)
• Compile and load a string as if it was a file for compile-file, and allow specification of source information and a compilation policy (swank-compile-string)
• Compile a file with a specified policy (swank-compile-file)

The compilation hooks are seemingly supposed to do a lot more than is really specified in the interface. Within call-with-compilation-hooks, compiler problems are supposed to result in a swank condition of type compiler-condition being signaled. compiler-condition has the following accessible:

• The original condition if there is one
• The "severity", which is one of :error :read-error :warning :style-warning :note :redefinition.
  • Read errors are handled separately because they indicate that the compiler absolutely cannot continue, as opposed to regular errors where it could continue from the debugger I guess? CLHS 3.2.5 is probably relevant here somehow.
  • Compiler notes are, in SBCL and Cleavir at least, a subtype of condition but not error or warning that the compiler sometimes spits out ot indicate that there's nothing wrong with the code per se but it still wants to tell you something. Note scould be for example optimization hints that are not necessarily portable and/or notices of compiler suboptimality.
• A brief message for display. Probably not relevant to a library.
• A string indicating the macroexpansion context of the code from which the problem originated.
• A list of references to standard documents.
• A source location (see below).

Gray streams

• Name of gray streams package (gray-package-name)
• Make a character output stream that calls a given function to output a string (make-output-stream)
• Make a character input stream that calls a given function to input a string (make-input-stream)

Introspection

Some of this is in Definitions.

• Get the lambda list for a given function name or function (arglist). Technically exposed in CL through function-lambda-expression.
• Get the function name of a function (function-name). Also technically available in f-l-e.
• Determine if a symbol is a type specifier (type-specifier-p)
• Determine if a given thing is a valid function name, accounting for extensions (function-name-p)
• macroexpand-all
• compiler-macroexpand-1, compiler-macroexpand. These are pretty easily written in portable CL already, I think.
• Get a list of all the macro or compiler macro subforms in a form (collect-macro-forms). A code walker should be more than adequate for this.
• Get a bunch of docstrings of extended type (describe-symbol-for-emacs). Possibly could just be a documented extension to documentation?
• Describe a symbol in a particular meaning, e.g. describe a symbol that names both a symbol and a function only as a symbol (describe-definition)
• Get a list of definitions of a symbol, with source locations (find-defintions)

Debugging

Lot of this is covered by Dissect. For some of the more active operators like restarting, maybe looking at Racket's continuations and prompts would be interesting.

• Install a debugger hook that BREAK and the stepper respect (install-debugger-globally, call-with-debugger-hook)
• Install an environment for a given debugger function, e.g. precompute backtraces (call-with-debugging-environment)
• Compute a backtrace, a list of frame objects (compute-backtrace). This might better be done within a context (like, call-with-backtrace) because a backtrace is inherently dynamic-extent.
• Get a list of references to standards corresponding to a given condition (condition-extras)

Frames

• Print human-readably (print-frame)
• Get a human-readable representation of the call (frame-call)
• Restart from a frame, i.e. conceptually unwind to the previous frame and call the function again with the same arguments (frame-restartable-p, restart-frame)
• Get the source location (frame-source-location)
• Get a list of active catch tags (frame-catch-tags)
• Get local variables and their values, with integer keys to disambiguate multiple available bindings with the same name (frame-locals, frame-var-value)
• Disassemble the code (disassemble-frame)
• Evaluate a form in the frame's lexical environment (eval-in-frame)
• Return the frame's... package? Not sure what that means. On SBCL it's the package of the function name (frame-package)
• Truncate the current continuation to the frame, then evaluate a form in the new continuation (return-from-frame). I think.

Stepping

An extension for this would probably involve a specified condition type for step conditions. I have done this in Clasp and it's usable in Jupyter through restarts for step-into etc.

• Prepare to step a frame (activate-stepping)
• Set a breakpoint when a frame returns (sldb-break-on-return)
• Set a breakpoint when a function with the given name is called (sldb-break-on-start)
• Determine whether a condition is a stepper condition (stepper-condition-p)
• Step into, over, out (sldb-step-into, sldb-step-next, sldb-step-out)

Source locations

This is its own thing instead of under a category because it's touched on by several categories (compilation, debugging, introspection). An extension for source locations would probably define a type for them with some accessors. Swank defines its own type to wrap whatever the implementation is doing, and expects functions that return a source location to use those objects instead of something native. So the only backend interface function is actually

• Locate an object (find-source-location)

but source locations are internally acccessed in Swank. This interface is a bit messy since it's tailored to Swank and so e.g. emacs buffers. But generally speaking source locations have or can have the following slots:

• Filename
• Character offset
• Line, column
• A function name, possibly with method specializers, which can be used to search a file for the definition
• Original source form (stringified)

Crossreference

Hopefully mostly pretty self-explanatory. Generally these return lists of "dspecs" with source locations. A dspec describes a definition, e.g. for a function it might be (DEFUN FOO). They're also returned by find-definitions. Most of these functions operate on names instead of actual objects, so e.g. who-calls accepts a function name but not a function.

Some of them have aliases and I don't know why.

who-calls ; a function
calls-who ; has exactly the same docstring as who-calls but does something quite different!
who-references ; a (presumably special?) variable
who-binds ; ditto
who-sets ; ditto
who-macroexpands ; a macro
who-specializes ; a class. seems portably implementable with mop:specializer-direct-methods + find-source-location?

list-callers and list-callees also exist as "lower level" variants. What this means interface-wise is not clear to me.

Profiling

No interface for profiling given a function object that I can see. That can be harder to implement, of course.

Interestingly SBCL only makes these available in its own interface as macros, so swank has to eval. Presumably the idea is an analogy to cl:trace.

There doesn't seem to be any separate interface for a statistical profiler.

• Profile, unprofile by function name (profile, unprofile)
• Get a list of currently profiled functions (profiled-functions). I think this returns names rather than functions.
• Unprofile all functions (unprofile-all)
• Report (profile-report)
• Clear all previously gathered statistics (profile-reset)
• Profile all functions with names in a package (profile-package). Doesn't specify if external only. Specifies that a previously profiled function is unprofiled and reprofiled but maybe that should be part of profile? Considers methods to be separate from discriminating functions, which again doesn't seem like a package specific concern. Also has an option to collect information about callers of these functions if available ditto.

Tracing

The only tracing interface is toggle-trace but it's a bit complicated. First, it's functional, unlike cl:trace. Secondly, it accepts extended designators of things to trace:

• Function names including setf functions (this is standard to cl:trace)
• (:defmethod name qualifier* (specializer*)) to trace a specific method
• (:defgeneric name) to trace a generic function and all its methods presumably separately, i.e. to note both the overall gf call and which specific methods were called.
• (:call caller callee) to trace only calls of callee by caller.
• (:labels toplevel local) to trace a local function within toplevel. (:flet toplevel local) is hopefully only supposed to be an alias.

SBCL's implementation of cl:trace additionally allows tracing compiler macros, and tracing all functions fbound to symbols in a given package.

SBCL, ECL, and Clasp (at least) support further options to cl:trace that can essentially make it a general tool for placing breakpoints or executing arbitrary code in and around given function calls:

• Custom call (:report in SBCL): Either NIL to not print anything (and therefore only do breaks etc), or call a given function with several parameters relating to the trace, e.g. the trace depth, function being called, if we're entering or exiting and how we're exiting, a stack frame, arguments or return values.
• Conditionality: Have trace do nothing unless a given arbitrary form evaluates to true on function entry or exit or both.
• Break: cl:break if a given form evaluates to true on function entry or exit or both.
• Stepping: Start stepping the function when it's called (this is in ECL/Clasp and I think is covered by the stepping interface earlier, except maybe the conditionality?)
• Print: Print the values of an arbitrary form along with the usual trace information.
• Wherein: Only trace if the function is being called by a given other function.
• Encapsulate: Controls whether tracing is done by redefining the function name or by actually altering the function. Not sure of the details.

Inspector

The main function here is emacs-inspect which is roughly to inspect as describe-object is to describe. Rather than doing the work of inspecting (i.e. presenting a REPL or whatever), this function returns a list specifying how to render the object for inspection. From the looks of it this defines a kind of very basic hypertext markup. Elements of the list can either by text to render the text, (:value something) to say there's a subobject to inspect, and (:action text function) to make a "link" that executes the function when clicked.

There's also eval-context, which I guess is supposed to return a list of bindings for let for evaluating within the context of inspecting a given object.

Concurrency

Bordeaux. Also my concurrency spec attempt that hasn't gone anywhere yet.

• initialize-multiprocessing
• Create a thread to call a given function with no arguments (spawn)
• Return a thread's name, a string (thread-name)
• current-thread
• Return a fresh list of all threads (all-threads)
• Determine whether a thread is running (thread-alive-p)
• Set the initial value for a special binding in new threads (set-default-initial-binding). IMO should work with thunks instead of forms
• Given a list of input (and in practice I think, file descriptor) streams, wait until one or more become ready, and return the list of those ready (wait-for-input)

Mailboxes

Each thread in Swank has its own queue that any thread can put stuff on concurrently but only it can read. These should be portably implementable in terms of basic concurrency things like locks and condition variables.

• Put an object in a thread's mailbox (send)
• Sleep until an object appears in this thread's mailbox. If it matches a given predicate, remove it from the mailbox and return it (receive-if). Also has a timeout.
• Ditto but with no predicate (receive)

Interrupts

Kind of vague.

• Interrupt a thread to have it call a function (interrupt-thread). Can be asynchronous and force the question. Impolite.
• Tell a thread to die asynchronously without calling any exit hooks like unwind-protect (kill-thread)
• Politely (not asynchronously) ask a thread to check for interrupts (wake-thread)

Locks

Though not documented, which bit me in the ass once, swank expects these locks to be recursive.

• Make a lock (make-lock)
• Hold a lock (call-with-lock-held)

Garbage collection

trivial-garbage covers these, I'm pretty sure.

• Make a weak-key hash table (make-weak-key-hash-table)
• Make a weak-value hash table (make-weak-value-hash-table)
• Return the weakness of a hash table, or NIL for non-weak hash tables (hash-table-weakness)

Things specific to swank, i.e. that aren't really a portability interface

default-readtable-alist
preferred-communication-style
emacs-connected
find-external-format ; arguable
gdb-initial-commands ; maybe?
thread-id find-thread ; I think these are only here for emacs, and in CL we'd just use the threads themselves
thread-attributes ; totally undefined result so I think it's only for display purposes
thread-status ; ditto, available as a real interface through thread-alive-p

Floating point

Should be covered by float-features

• float-nan-p
• float-infinity-p

Images

UIOP does this I think?

• Save lisp to a file and die (save-image). Accepts a function that the image should call on startup.
• Do that but in the background (background-save-image)

Advice

• Given a function name (?) wrap future calls to it in callbacks (wrap)
• Remove previously specified advice (unwrap)
• Check if a function name has advice (wrapped-p)

Things I don't understand

make-auto-flush-thread
print-condition
buffer-first-change
describe-primitive-type
register-thread find-registered ; seems to only be used for the sentinel. should be implementable by holding the thread and using CAS or something to prevent dupes
character-completion-set ; has a matchp argument that's a function. is that explained? no.

Miscellaneous

• quit-lisp. Portably available in the shut-it-down library.
• lisp-implementation-type-name. Not sure how this is distinct from cl:lisp-implementation-type.
• call-with-syntax-hooks. "Call FN with hooks to handle special syntax." I don't know what that means, but it handles "sb!" package names.
• guess-external-format. Given a pathname try to guess what external format is appropriate for input.
• format-string-expand. This seems like it should portably be macroexpand of formatter, so I don't know why there's an interface. Some old crappy nonconforming Lisp?