Whole project loading for Haskell IDE tooling

From Haskell Language Server 2.15.0.0 onwards, users can opt-in to loading all components of their current project up front, rather than as needed. Doing so ensures information about every module is fully available from the start, and avoids pauses to re-initialize the session when opening a file from a new component.

The feature is enabled by setting componentsLoading to "multi: whole-project" in the HLS configuration.

The next release of Haskell Debugger will load all components up front too. Debug sessions will then be able to support breakpoints from any module in the project, not just the entry point’s component.

The feature is provided by hie-bios >= 0.20 and the cradle config (hie.yaml) can be used to fine-tune the list of components to load. The new componentsToLoad field lets you list the exact targets to pick, which are otherwise the components listed in the cradle or inferred from the cabal/stack project.

Multi component sessions

Haskell Language Server has supported multi component sessions for a while, e.g. when working on a Cabal package with both a library and an executable their files can all be loaded together, so jumping to definition from app/Main.hs can land you in lib/Foo.hs.

However, so far each new component has been loaded into the session lazily, i.e. only when files belonging to the component are opened. There are caches involved, but each time this happens the GHC session is re-initialized from scratch: the new set of components, and how to build them, is discovered by querying the underlying build system, e.g. by invoking cabal repl/stack repl and extracting the flags they would pass to ghci. Ultimately the new component might have to go from being pre-compiled on disk to interpreted in-memory, forcing all its descendants to be reconfigured too, so even if we wanted to be more clever not much work could be saved.

This is not a big deal with just two components, but in a larger project it can be surprising to have HLS behave differently in a given file according to which other files have been opened. Also, when the session needs to be re-initialized because of a new component, the induced pause can be a bit jarring.

Whole project loading

The new whole project mode instead loads all project components right away, so that everything is available from the start. The downside is doing everything up front means memory use and initialization time are also paid up front. Also, a misconfigured component can stop the others from loading.

In Haskell Language Server

Eventually we hope to make whole project loading the default. For the moment, starting with haskell-language-server-2.15, the new feature is enabled by setting the new config field componentsLoading to multi: whole-project. The other options are multi: needed-only, which is the default, and single, which loads only a single component at once. The componentsLoading field replaces the previous sessionLoading one which is now deprecated: sessionLoading: "multipleComponents" is now componentsLoading: "multi: needed-only" and interpreted as such.

In Haskell Debugger

Haskell Debugger was actually the primary motivation for this work, as finding out in the middle of a debug session that you cannot add a breakpoint in an imported module is more dire than the issues with HLS.

Prior to whole-project loading, the typical setup where a package contains both a library and an executable that depends on it would end up with only the modules from the executable interpreted and so available to be stopped at. The library was installed in a temporary package database and included only as a dependency, so trying to set a breakpoint in a module from the library would just fail.

Haskell Debugger builds its session only once, as soon as it learns what the entry point is, i.e. the module and function to start execution from. At this point the debugger doesn’t know where the breakpoints will be, and so which modules need to be interpreted. As new breakpoints can be set while execution already started, waiting to know all of them to initialize the session is not an option.

Because of the above we have gone ahead and made whole-project loading the default and only way Haskell Debugger behaves, starting from the next release.

How it works and further configuration

What “all project components” means depends on the underlying cradle, either specified in a hie.yaml file or inferred from what can be found in the root directory. Recent enough Cabal and Stack are the two kinds of cradles that actually support this: if the hie.yaml file specifies components we load those, otherwise we ask the package manager to infer them. Inferring means asking Cabal for target all while enabling tests and benchmarks, or asking Stack for its default targets.

The above logic, like most of this new feature, is implemented by hie-bios starting from version 0.20. All details on how to fine-tune your cradle can be found in hie-bios’s README, including the new componentsToLoad field.

Before explaining the new field let’s go through a few examples with Cabal cradles to make things more concrete. For each case we will list the cabal repl invocation that hie-bios uses to fetch the correct flags.

A very minimal cradle (hie.yaml) will let cabal infer the components by using the target all, i.e.

cradle:
  cabal:

will result in cabal repl --enable-multi-repl --enable-tests --enable-benchmarks all. The --enable-* flags are included so that the session will cover everything in the project.

The --enable-* flags will override what specified in a cabal.project file, so if the cradle includes one we omit them, i.e.

cradle:
  cabal:
    cabalProject: "./cabal.project"

will result in cabal repl --enable-multi-repl --project-file=./cabal.project all.

If components are specified then those are the targets used, e.g.

cradle:
  cabal:
    components:
      - {"path": "./src", "component": "lib:foo"}
      - {"path": "./app", "component": "exe:foo"}

will result in cabal repl --enable-multi-repl lib:foo exe:foo.

Finally, if some component doesn’t play nice with cabal repl or the tooling that uses hie-bios, the field componentsToLoad can be used to specify the exact list of targets to use, for example

cradle:
  cabal:
    componentsToLoad: ["lib:foo","lib:bar"]

will result in cabal repl --enable-multi-repl lib:foo lib:bar.

Conclusion

Hopefully whole-project loading will provide a less surprising and smoother dev experience on projects with multiple components. It’s been my new default and working well for me since it got merged. Feedback is welcome on the HLS issue tracker. Issue #5005 is the issue about the feature introduction.

This work has been performed in collaboration with Mercury, who have a long-term commitment to the scalability and robustness of the Haskell ecosystem. Well-Typed are always interested in projects and looking for funding to improve HLS and other Haskell tools. Please contact info@well-typed.com if we might be able to work with you!