DRAFT jupyter(lab)-lsp incorporation pre-proposal

preproposal.md

Pre-Proposal Checklist

• Briefly outlines the proposal
• Suggests a review team (optional)
• Declares why it should be a JEP (See the “JEP / Not-a-JEP Rubric” below.)
• Identify a Shepherd to see the process through. (Shepherds are assigned on a round-robin basis from a set of interested engaged volunteers).
• Decide if it’s a JEP according to the criteria listed above. The Shepherd decides if the JEP criteria have been met.

Pre-Proposal

The maintainers of jupyter(lab)-lsp would like to propose its incorporation as an official sub-project of Project Jupyter. We feel this would benefit Jupyter users through better discoverability of advanced interactive computing features, supported by the Language Server Protocol (LSP), but otherwise missing in a user's Jupyter experience.

The key component of the repository, @krassowski/jupyterlab-lsp, offers Jupyter users an expanded subset of features described by the LSP as an extension to JupyterLab. These features include refinements of existing Jupyter interactive computing features, such as completion and introspection, as well as new Jupyter features such as linting, reference linking, and symbol renaming. It is supported by jupyter-lsp, a Language Server- and Jupyter Client-agnostic extension of the Jupyter Notebook Server (for the 0.x line) and Jupyter Server (for the 1.x). We will discuss the architecture and engineering process of maintaining these components at greater length, leveraging a good deal of the user and developer documentation.

Much like Jupyter Kernel Messaging, LSP provides a language-agnostic, JSON-compatible description for multiple clients to integrate with any number of language implementations. Unlike Kernel Messaging, the focus is on precise definition of the many facets of static analysis and code transformation, with nearly four times the number of messages of the Jupyter specification. We will discuss the opportunities and challenges of this complexity for users and maintainers of Jupyter Clients, Kernels, and related tools.

Branding

We invite discussion on an appropriate Jupyter branding/organization approach, ranging from:

• a single move of the @krassowski/jupyterlab-lsp monorepo to @jupyter/lsp
• a new, Jupyter co-branded GitHub organization (e.g. jupyter-lsp)

Kernel integration

In order for jupyterlab-lsp to be able to connect to the appropriate Language Server in tandem with a Kernel, it requires that kernels properly implemenet KernelInfo from version 5.0 of Jupyter Messaging specification, in particular the field language_info with the following fields present:

• mimetype
• file_extension
• name

Therefore, no changes to the specification are required. Though some future specification refinements could be discussed as separate JEPs.

In continuous integration, we keep under test Kernels and Language Servers for all three of the Jupyter-name forming languages:

• Julia: IJulia using LanguageServer.jl,
• Python: IPython using pyls or jedi-language-server,
• R: IRKernel using R languageserver

Other kernels also confirmed to work straightaway (Bash) or after minimal user configuration (Scala) in presence of appropriate Language Servers. Academic prototype works were created using the LSP as a connector.

Magics and polyglot support

We also support the magics in IPython, allowing for polyglot experience when using cell magics. For the detailed discussions on magics adoptions please refer to https://github.com/krassowski/jupyterlab-lsp/issues/347.

Outlook towards notebook-specific IDE features

The team envisions a support for the notebook specific IDE features to be included in the future. This might involve upstream changes to the LSP specification, or custom jupyter-specific extension. Please see our community discussion for more details.

Minimal implementation details

• jupyterlab-lsp implements integration with JupyterLab and its default CodeMirror editor; the design allows for incorporation of additional editors in the future, but large portions of code are CodeMirror 5-specific; it communicates with jupyter-lsp to send/recieve messages from language servers and with JupyterLab to get information about currrent kernel
• jupyter-lsp uses jupyter_server, tornado, sockets and IO streams for communication with servers and with frontend; it is not aware of a kernel's existence
• we use GitHub CI running pytest, jest, and robot (Selenium) test; the integration/acceptance tests for the bulk of the frontend testing; we test on Windows, Linux and MacOS, supporting thee Python versions at a time for jupyter-lsp and doing tests with actual language servers

Previous discussions:

Please refer to:

• jupyterlab/frontends-team-compass#67 for the discussion held within the JupyterLab team.
• jupyter/enhancement-proposals#26 for previous (stale) JEP on LSP adoption; this proposal superseds that JEP; importantly jupyter(lab)-lsp does not require any specific editor, whereas the previous JEP specifically mentions Monaco.