Skip to content

Instantly share code, notes, and snippets.

@danidiaz

danidiaz/_Haskell_Cabal_hacking_notes.md

Last active June 15, 2021 17:49
Show Gist options

  • Select an option

    Learn more about clone URLs
  • Save danidiaz/4345c9d93ef7b64368bc7fdee72c5252 to your computer and use it in GitHub Desktop.

Select an option

Learn more about clone URLs
Save danidiaz/4345c9d93ef7b64368bc7fdee72c5252 to your computer and use it in GitHub Desktop.
Download ZIP
Haskell Cabal hacking notes
Raw
_Haskell_Cabal_hacking_notes.md

Cabal hacking notes

Raw
commands.md

For compiling cabal install:

cabal build --jobs=2 --allow-newer="base" --allow-newer="template-haskell" cabal-install:exe:cabal

Testing my example with the compiled cabal-install:

cabal test --verbose=3 moo-oops

Looking for some text in files which have some import:

rg -l 'Distribution.Simple.Configure'  | xargs rg '[^\w]configure[^\w]'

Use --project-file=cabal.project.release? see also. seems that the annoyance has been corrected.

trouble with Nix integration

cabal build --dry --upgrade-dependencies --allow-newer

Finding the generated cabal-install executable after a compilation:

cabal list-bin cabal-install:exe:cabal
Raw
issues.md

Issue #7420

Improve debug logging in Cabal/cabal-install

*** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace:
  Distribution.Simple.Utils.topHandlerWith.handle,
  called from Distribution.Client.ProjectPlanOutput.writePlanExternalRepresentation,
  called from Distribution.Client.ProjectPlanning.rebuildInstallPlan.phaseMaintainPlanOutputs,
  called from Distribution.Client.RebuildMonad.liftIO,
  called from Distribution.Client.RebuildMonad.rerunIfChanged,
  called from Distribution.Client.RebuildMonad.runRebuild,
  called from Distribution.Client.ProjectPlanning.rebuildInstallPlan.\,
  called from Distribution.Client.ProjectPlanning.rebuildInstallPlan,
  called from Distribution.Client.ProjectOrchestration.runProjectPreBuildPhase,
  called from Distribution.Client.CmdBuild.buildAction,
  called from Distribution.Simple.Command.commandAddAction.applyDefaultArgs,
  called from Distribution.Simple.Command.fmap,
  called from Distribution.Simple.Command.commandAddAction,
  called from Distribution.Client.CmdLegacy.newCmd.cmd,
  called from Distribution.Client.CmdLegacy.newCmd,
  called from Main.mainWorker.commandSpecs,
  called from Distribution.Simple.Command.commandFromSpec,
  called from Main.mainWorker.commands,
  called from Distribution.Simple.Utils.topHandlerWith,
  called from Distribution.Simple.Utils.topHandler,
  called from Main.mainWorker,
  called from Main.main

At the top level, top-level actions like Distribution.Client.CmdBuild.buildAction are not-too-weird functions, and seem amenable to instrumentation...

Nice to have:

  • It should not require big changes to the current internal structure and implemenation of the main functions.
  • It should not force a "big-bang" global change; functions should be able to opt into the "instrumentation" system one by one.
  • It should not force functions to take large numbers of additional positional parameters (they have a good amount of them already!)
  • The structure of import dependencies between modules should stay largely unchanged. Modules should not depend on some "central composition module" that imports everything else. (Such a "central compositions module" might exist, it's just that most other modules shouldn't be aware of its existence.)
  • Functions shouldn't care about the way their direct dependencies (let alone transitive ones) have been instrumented. For example, if a function f calls a function g, and g has been instrumented to send their arguments to a separate logging file, f should be completely ignorant about that file, and should not have to pass around the file handle to g.
  • It should not force programmers to manually add instrumentation for each function parameter.
  • We should be able to instrument auxiliary functions defined in let and where clauses, not only top-level functions.

Stuff:

  • CompositionContext, cc for short.
  • Has typeclass in the prelude.
  • Instrumentable typeclass.
  • Instrumeter record (how to make it variadic?). makeInstrumenter ? Idea: Instrumentable instance for Instrumenter that passes around the intrumentation function.
  • ToJSONDebug typeclass?

In case the Generic instances became a problem.

This bit about how gc works might be relevant

I guess LogProgress will have to be turned into a cabability as well...

This is worrying:

 src/Distribution/Compat/Semigroup.hs:46:43: warning: [-Wderiving-defaults]
      • Both DeriveAnyClass and GeneralizedNewtypeDeriving are enabled
        Defaulting to the DeriveAnyClass strategy for instantiating Binary
        Use DerivingStrategies to pick a different strategy
      • In the newtype declaration for ‘Last'’
     |
  46 |   deriving (Eq, Ord, Read, Show, Generic, Binary, Typeable, Inspectable)
     |

Maybe write manual instances?

Idea: dont' import the custom Prelude from Instrumentable. Instead of that, make the custom prelude import Instrumentable. Continue providing instances for types like NonEmptySet in Instrumentable.

import Distribution.Client.Utils.Inspectable (Inspectable)

a proof-of-concept solution

Further idea: when you are in a hurry and don't want to give Inspectable instances to the arguments and return type of your bean, but you still want the bean to take part in dependency injection, wrap it in an Opaque newtype that has a special Instrumentable instance which discards all instrumentations. Meanwhile, the Has instance could make use of coerce to reduce boilerplate.

This can change the has cc into self:

class Has r cc where
    has :: cc -> r

newtype Self cc = Self {self :: forall r. Has r cc => r}

selfie :: cc -> Self cc
selfie cc = Self (has cc)

Also works with ViewPatterns.

Example:

 makeBuildAction_ (Self {self}) ...
     ...
     runProjectPreBuildPhase self verbosity baseCtx

However, using Self at a global level seems too confusing and laborious for little gain. Better use it only at function level.

Problems with Inspectable instances:

Missing instance:

No instance for (GInspectableFields U1)

Not knowing if a generic parameter is a list or not.

Instance for maps.

Instance for internal types (will perhaps require moving Inspectable into Cabal.)

Compare the two JSON implementations.

instance Show Program where
  1   show (Program name _ _ _ _) = "Program: " ++ name
  2
  3 instance Inspectable Program

This is tricky, for NubList:

instance InspectableString (IsTheElementChar a) a => Inspectable (NubList a)

Besides Inspectable, there's Show, Pretty, Structured...

  instance Binary VerbosityFlag
  instance Structured VerbosityFlag
  instance Inspectable VerbosityFlag

Perhaps put Inspectable in each respective prelude?

Interesting: Graph has a special non-generic Structured instance.

Issue #7394

shell hooks for augmenting how cabal does various actions?

Issue 6835

Unable to build with backpack modules from unpacked-containers package

compLinkedLibDependencies, that's the important field from mixin linking

  • in function configureComponentLocalBuildInfos, there is a progression:

    ConfiguredComponent -> LinkedComponent -> ReadyComponent

  • Where are instantiated libraries installed when...

    • ... the indefinite library and the implementation library are both external?
    • ... the indefinite library and the implementation library are both local?
    • ... the indefinite library is external and the implementation library is local?
    • ... the indefinite library is local and the implementation library is external?

  • compiling the example that works:

    /home/daniel/hs/cabal/dist-newstyle/build/x86_64-linux/ghc-9.0.1/cabal-install-3.4.0.0/x/cabal/build/cabal/cabal +RTS -xc -RTS build moo-nad-x:tests --verbose=3 > /tmp/logz/logz.txt

  • compiling the example that doesn't work:

    /home/daniel/hs/cabal/dist-newstyle/build/x86_64-linux/ghc-9.0.1/cabal-install-3.4.0.0/x/cabal/build/cabal/cabal +RTS -xc -RTS build tests --verbose=3 > /tmp/logz/logz.txt

  • looks at the pkgRoot field of InstalledPackageInfo to see if it's in dist-newstyle or in the store.

  • Cabal.Distribution.Simple.Configure.configure which is set as confHook in Cabal.Distribution.Simple.

  • traced to this configureCommand.

  • perhaps looks straight into Distribution.Client.ProjectPlanning, Distribution.Client.ProjectConfig and Distribution.Client.ProjectBuilding?

  • Distribution.Client.ProjectPlanning.instantiateInstallPlan.

    NB: elab is setup to be the correct form for an indefinite library, or a definite library with no holes. We will modify it in 'instantiateInstallPlan' to handle instantiated packages.

    instantiateComponent

dist_dir = distBuildDirectory distDirLayout
                  (elabDistDirParams elaboratedSharedConfig elab)

Distribution.Client.DistDirLayout.

Distribution.Client.ProjectPlanning: pkgsToBuildInplaceOnly, pkgsLocalToProject.

shouldBuildInplaceOnly :: SolverPackage loc -> Bool
shouldBuildInplaceOnly pkg = Set.member (packageId pkg)
                                        pkgsToBuildInplaceOnly

pkgsToBuildInplaceOnly :: Set PackageId
pkgsToBuildInplaceOnly =
    Set.fromList
  $ map packageId
  $ SolverInstallPlan.reverseDependencyClosure
      solverPlan
      (map PlannedId (Set.toList pkgsLocalToProject))

Is this done before instantiateInstallPlan? Then we have a problem.

It seems all components go through instantiateComponent, but non-backpack components do so in a trivial way. There must be some place at which "new" components are created for indefinite componets which are instantiated.

 Distribution.Client.RebuildMonad.liftIO,
called from Distribution.Client.RebuildMonad.>>,
called from Distribution.Client.RebuildMonad.rerunIfChanged,
called from Distribution.Client.RebuildMonad.runRebuild,
called from Distribution.Client.ProjectPlanning.rebuildInstallPlan,
called from Distribution.Client.ProjectOrchestration.runProjectPreBuildPhase,
called from Distribution.Client.CmdTest.testAction,
called from Distribution.Simple.Command.commandAddAction,
called from Distribution.Client.CmdLegacy.newCmd,
called from Distribution.Simple.Command.commandFromSpec,
called from Distribution.Simple.Utils.topHandlerWith,
called from Distribution.Simple.Utils.topHandler,
called from Main.mainWorker,
called from Main.main

Instantiations are performed before the "improvement". After "Elaborating the install plan..." but before "Improving the install plan..."

Issue 7423

cabal check warns about -O2 even if it is just a flag

CheckTests.

    checkGhcOptions "ghc-options" (hcOptions GHC) pkg

Distribution.Types.BuildInfo the options field.

7394

shell hooks for augmenting how cabal does various actions?

          "projectConfigHcPath": {
            "Flag": [
              "ghc-8.10.4"
            ]
          },

which is part of ProjectConfigShared (shared between all packages in a project).

(dbg) Entered buildAction.
(dbg) Entered runProjectPreBuildPhase.
(dbg) Entered rebuildInstallPlan.
cabal: Cannot find the program 'ghc'. User-specified path 'ghc-8.10.2' does
not refer to an executable and the program is not on the system path.

$ cabal configure
cabal: Cannot find the program 'ghc'. User-specified path 'ghc-8.10.2' does
not refer to an executable and the program is not on the system path.

Look at configureProgram

It seems that with cabal 3.5, configure doesn't invoke preBuild:

commit e70b5eb9df8f5472664ab779b432d305b55f7656
Author: Patrick Augusto <[email protected]>
Date:   Thu Apr 22 15:50:44 2021 -0300

    cabal v2-configure, see issue #7405

    This commit straightens the v2-configure command:

    * Removes the pre-build phase
    * Adds two flags, --append and --overwrite

    Co-authored-by: Emily Pillmore <[email protected]>

Distribution.Compat.Process and Distribution.Client.Compat.Process. The latter exposes readProcessWithExitCode.

Interesting syntax in record construction:

ProjectConfigShared {..} = defaults <> projectConfigShared
Raw
notes.md

  • Contributing to Cabal

  • about the Cabal testsuite

  • Cabal the library CHANGELOG

  • cabal-install CHANGELOG

  • what can be used?

  • idea for debugging: compile with profiling to have a way of inspecting stack traces.

     ghc-options: -fno-ignore-asserts -prof -fprof-auto

    alas:

    Failed to load interface for ‘GHC.Num.BigNat’ Perhaps you haven't installed the "p_dyn" libraries for package ‘ghc-bignum’

    -prof GHC flag should not to be used with --ghc-options cabal flag

    the -xc RTS flag so perhaps invoke it like +RTS -xc -RTS?

    It seems that it's enough to add the following to cabal.project:

    profiling: True

    And then invoke cabal with

    cabal +RTS -xc -RTS ...

    Here's a nice example stack trace:

      cabal: Failed to build 
  moo-nad-0.1.0.2-136596e5d30785f22eda9db329422a145f8a7b99b9390088a3356b95680ad70e+CXHgquAbAM34wnUHImMH6r.
  The failure occurred during the configure step.
  Failed to build moo-nad-x-0.1.0.2 because it depends on moo-nad-x-0.1.0.2
  which itself failed to build.
  Failed to build moo-nad-x-0.1.0.2 because it depends on moo-nad-x-0.1.0.2
  which itself failed to build.

  *** Exception (reporting due to +RTS -xc): (THUNK_STATIC), stack trace:
    Distribution.Simple.Utils.die',
    called from Distribution.Client.ProjectOrchestration.dieOnBuildFailures,
    called from Distribution.Client.ProjectOrchestration.runProjectPostBuildPhase,
    called from Distribution.Client.CmdTest.testAction,
    called from Distribution.Simple.Command.commandAddAction,
    called from Distribution.Client.CmdLegacy.newCmd,
    called from Distribution.Simple.Command.commandFromSpec,
    called from Distribution.Simple.Utils.topHandlerWith,
    called from Distribution.Simple.Utils.topHandler,
    called from Main.mainWorker,
    called from Main.main

  • idea for debugging: derive Generic for interesting objects and then bring in "generic-lens".

  • Distribution.Client.ProjectPlanOutput.writePlanExternalRepresentation might be useful for debugging.

    The auxiliary elaboratedPackageToJ function doesn't seem to print the Backpack ModuleShape.

  • cabal-install has integration tests in /tests.

  • Distribution.Client.ProjectPlanning.Types.ElaboratedInstallPlan.

  • Distribution.Client.ProjectPlanning.elaborateInstallPlan has Backpack-related stuff.

    Two variants of the install plan are returned: with and without packages from the store. That is, the "improved" plan where source packages are replaced by pre-existing installed packages from the store (when their ids match), and also the original elaborated plan which uses primarily source packages.

    Update the files we maintain that reflect our current build environment. In particular we maintain a JSON representation of the elaborated install plan (but not the improved plan since that reflects the state of the build rather than just the input environment).

    Improve the elaborated install plan. The elaborated plan consists mostly of source packages (with full nix-style hashed ids). Where corresponding installed packages already exist in the store, replace them in the plan.

    The improved plan changes each time we install something, whereas the underlying elaborated plan only changes when input config changes, so it's worth caching them separately.

    data OpenUnitId
        -- | Identifies a component which may have some unfilled holes;
        -- specifying its 'ComponentId' and its 'OpenModuleSubst'.
        -- TODO: Invariant that 'OpenModuleSubst' is non-empty?
        -- See also the Text instance.
        = IndefFullUnitId ComponentId OpenModuleSubst
        -- | Identifies a fully instantiated component, which has
        -- been compiled and abbreviated as a hash.  The embedded 'UnitId'
        -- MUST NOT be for an indefinite component; an 'OpenUnitId'
        -- is guaranteed not to have any holes.
        | DefiniteUnitId DefUnitId
      deriving (Generic, Read, Show, Eq, Ord, Typeable, Data)
    -- TODO: cache holes?
    
        -- OpenModule

    -- | Unlike a 'Module', an 'OpenModule' is either an ordinary
    -- module from some unit, OR an 'OpenModuleVar', representing a
    -- hole that needs to be filled in.  Substitutions are over
    -- module variables.
    data OpenModule
        = OpenModule OpenUnitId ModuleName
        | OpenModuleVar ModuleName
      deriving (Generic, Read, Show, Eq, Ord, Typeable, Data)  

-- | A 'ModuleShape' describes the provisions and requirements of
-- a library.  We can extract a 'ModuleShape' from an
-- 'InstalledPackageInfo'.
data ModuleShape = ModuleShape {
    modShapeProvides :: OpenModuleSubst,
    modShapeRequires :: Set ModuleName
    }

Does the above mean that shapes can only be truly known when a package is installed?

-- | An explicit substitution on modules.
--
-- NB: These substitutions are NOT idempotent, for example, a
-- valid substitution is (A -> B, B -> A).
type OpenModuleSubst = Map ModuleName OpenModule

       -- | This is true if this is an indefinite package, or this is a
       -- package with no signatures.  (Notably, it's not true for instantiated
       -- packages.)  The motivation for this is if you ask to build
       -- @foo-indef@, this probably means that you want to typecheck
       -- it, NOT that you want to rebuild all of the various
       -- instantiations of it.
       elabIsCanonical :: Bool,

Distribution.Client.ProjectBuilding.rebuildTargetsDryRun

mkDefUnitId is the function which creates the unit ids with the +.

type InstS = Map UnitId ElaboratedPlanPackage
type InstM a = State InstS a

Computations in InstM do a lot of knot-tying.

The result of functions like instantiateUnitId is not used (execState, for_).

danidiaz: Hi. I have a question about cabal-install. While building a project, it seems that, after calculating a suitable install plan, the executable calls itself with the configure sub-command. This is for building individual packages. Is my impression correct?

danidiaz: I got that impression from here: haskell/cabal#6835 (comment)

fgaz: danidiaz: almost: cabal-install builds and calls Setup.hs (that is linked against Cabal the library) in case of build-type: Custom

fgaz: in case of build-type: Simple, it just uses the built-in Cabal

fgaz: => it just calls a function from the Cabal library with the appropriate arguments

fgaz: Cabal takes care of building individual packages, while cabal-install takes care of the dependencies and build environment in general

-- The plan for what to do is represented by an 'ElaboratedInstallPlan'

-- Now given the specific targets the user has asked for, decide
-- which bits of the plan we will want to execute.
--
(elaboratedPlan', targets) <- selectPlanSubset elaboratedPlan


-- Concurrency control: create the job controller and concurrency limits
-- for downloading, building and installing.
jobControl    <- if isParallelBuild
                   then newParallelJobControl buildSettingNumJobs
                   else newSerialJobControl
registerLock  <- newLock -- serialise registration
cacheLock     <- newLock -- serialise access to setup exe cache
                         --TODO: [code cleanup] eliminate setup exe cache
                         
buildAndInstallUnpackedPackage

From Distribution.Client.InstallPlan:

-- The goal is to calculate an installation plan that is closed, acyclic and
consistent and where every configured package is valid.

  -- Only need Configured; this phase happens before improvement, so
  -- there shouldn't be any Installed packages here.

From ?

5 -- Source package have somewhat flexible dependencies. They are specified as 1 -- version ranges, though really they're predicates. To make matters worse they 2 -- have conditional flexible dependencies. Configuration flags can affect which 3 -- packages are required and can place additional constraints on their 4 -- versions.

Interesting bit about the efficiency of plan.json:

@hvr spent considerable time to keep the critical path to produce a plan.json fast, so that a no-op cabal build and cabal run latency would be <100ms in order to keep unconditional cabal build and cabal run usable in interactive settings or when integrating with other build tools.

Proposal for a Cabal plugin API (inversion of control) (2015)

From Distribution.Simple.Utils:

We must make all logging formatting and emissions decisions based on the 'Verbosity' parameter, which is the only parameter that is plumbed to enough call-sites to actually be used for this matter. (One of Cabal's "big mistakes" is to have never have defined a monad of its own.)

Raw
tests.md
  • PackageTests/Backpack/T6385/cabal.test.hs : withShorterPathForNewBuildStore.
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment