The Library Problem

the_library_problem.md

The Library Problem

General overview of the library problem.

Note: The terms library, module, and package are used interchangeably. Note: This document is subject to extension and modification

Implicit Assumptions:

• It must be in keeping with the general language's design and philosophy
• There should only be one way to do things and make that one way very well polished
  • Pragmatic, minimalist, and "right tool for the right job" philosophy
  • Prefer a knife over a Swiss Army Knife when you need a knife
  • Only allow multiple ways when absolutely necessarily either due to sanity of life or semantic necessity
• There should be such thing as a library
• There should be no need for a preprocessor
• Prevent namespace collision between libraries
• Deterministic linking names for entities
• There is a file system, made from files and directories
• Importing produces an import name to refer to that library
• Source code is written in files
• Flatter hierarchies are ideal (no nesting of namespaces)
• Out of order entity checking
• No procedure prototypes
• No extra files containing metadata, the source is all that is required
  • One language, not many
• When importing, imports an entire library and not individual entities from the library
• Imports are local to a file/package and not exported on subsequent imports unless explicitly specified (Thinking locally about a problem)
• Platform specific code in libraries must be possible and simple
• Imports should only be allowed as the file scope

General wants:

• Optimally no cyclic dependencies/imports
• Ability to have "library collections" (explicit search paths)
• Not require file extension if present
• When statements (compile-time if statement) at file scope to import platform specific packages
• Single file libraries in the style of Sean Barrett

General Conclusions:

• A package must be focused on either a file or a directory
  • A hybrid is possible but will focus one aspect more than the other
  • Hybrids will produce weird and inconsistent organizational issues
• The reason many package managers exist is because the concept of a package is ill-defined (not always the case, in some cases, the language is designed around having a package manager in the first place)
• You cannot all these assumptions and wants at once, you must compromise
• Having file scope when statements (compile-time if statements) means you cannot have cyclic imports/dependencies
  • Causes extremely difficult to solve (slow compile times) or ambigious (indeterminiate) import graphs
• To have deterministic linking names for entities either requires:
  • Explicit use defined prefix (package name) for each package
  • Use the directory path from a base directory and generate a prefix from that
• Implicit procedure overloading will be limited to a per scope basis unless there are extremely complicated look up rules
• Operating overloading requires the idea of a trait, as non-global (package-level) scopes complicate layout
  • Could be solved by:
    • Not having operator overloading
    • Making all operator overloads global scope and not allowing nested operator overloading (TODO: need to expand more on this as it is not obvious why this is true)

File based solutions (Python-like):

• If each file is a scope, splitting a package across multiple files may require another parent file which certain entities can be shared from
• If each other file is "#include"d, preventing other imports/includes of those files is necessary
• To treat a directory as a scope, something like a special file (e.g. __init__.py) will be needed
• From experience, people import single entities into a file which means it becomes harder to know where things are imported from and it comes a cluster-fuck to organize and reason with
  • Produces a very hierarchical situation

Directory based solutions (Go-like):

• Requires a directory for each package
• All files within a package can easily refer to one another preventing a hierarchy
• The file name itself can specify platform information foo_windows_amd64.odin
• Small packages (single file) require a completely new directory
• Packages that require other data (.lib, assembly, etc) can be contained within the directory