Well-Typed has been collaborating with the Haskell Language Server (HLS) development team thanks to funding from Mercury, Hasura and the HLS Open Collective to support HLS development. This includes work on performance, release management and support for newer GHC releases, as well as taking advantage of new GHC features such as Multiple Home Units and serializing Core to improve performance.
HLS releases
HLS is heavily tied to GHC and the GHC API, so to use HLS with a Haskell project, you must use an HLS executable compiled with the exact same GHC that you use to compile your project. Not doing so can often lead to strange and inscrutable errors.1 Thus HLS and the GHCup installation tool distribute a matrix of HLS binaries that exactly match the corresponding GHC binaries installed by GHCup. To ensure that HLS binaries corresponding to new GHC releases are promptly available for developers to use, we must provide new HLS releases whenever new major or minor releases of GHC are made.
Thanks to the generous donations of the Haskell Community to the HLS Open Collective and in collaboration with Julian Ospald and the HLS maintainers, Well-Typed has been working to ensure that new HLS releases promptly follow the GHC release cycle, that the release CI responsible for actually producing HLS binary distributions is robust and easy to use, and that the metadata necessary for GHCup to provide HLS binaries is updated along with the release.
This has involved producing HLS releases including the 1.7.0.0, 1.8.0.0, 1.9.0.0, 1.10.0.0 and 2.0.0.0 HLS releases. For the future, we hope to encourage volunteers to create more releases, while ensuring that someone from Well-Typed is always available as a backup to ensure timely releases.
HLS support for new major GHC versions
HLS is a big and complex project with many sub packages and dependencies, and is a prominent client of the notoriously unstable GHC API. As such, it can be quite a task to upgrade it to work new GHC versions.
As part of our work on HLS, we also use it as a staging ground for features that eventually make their way into GHC, such as serializing Core to improve performance. We also implement GHC features to make HLS work better (such as Multiple Home Units). These also require adapting so that HLS can work seamlessly with newer or older GHC versions.
Well-Typed has worked to upgrade HLS to have support for the GHC 9.2, 9.4 and
9.6 release series.
We have also added ghcide (the core of HLS) to
head.hackage to ensure that
it is kept up to date with changes in GHC, and to make future porting efforts
easier.
This work was supported by a combination of funding from the HLS Open Collective
(for 9.4 support for HLS plugins, in particular adapting to various
ghc-exactprint changes), Hasura (for 9.2 and 9.4 and Multiple Home Unit support)
and Mercury (for 9.6 support).
Of course, this work would be impossible without the help of the many volunteers who contribute to HLS as well as all the maintainers and contributors to the many packages and dependencies that HLS relies upon.
Performance and memory usage improvements
We previously implemented improvements to recompilation avoidance and startup time as part of our ongoing work for Mercury. Recently, we have made efforts to reduce the memory usage of HLS so that it is more feasible to use it on larger projects and memory constrained systems.
While HLS builds on the GHC API, it faces a different set of design constraints than a compiler like GHC which is traditionally used in batch mode. Rather than being a fire and forget program, HLS sessions are typically quite long lasting. Additionally, HLS is optimised for interactive use, aiming to provide low latency results to assist developers in a timely fashion as they write their programs. It is also heavily incremental, resuing old results as much as possible rather than restarting compilation from scratch, and making use of out of date results to provide low latency information even at the cost of not always being strictly correct.
As such, it needs to keep track of a lot of additional information for these purposes that a typical GHC or even GHCi session does not, which has a cost in terms of memory usage. The memory usage of HLS has been a bone of contention for a while on large projects, which is why Mercury asked us to investigate and reduce the memory usage.
Through a combination of profiling HLS using info table profiling along with careful investigation using ghc-debug, we have managed to significantly reduce the memory used by HLS, and make further improvements to the startup time. Check out Finley’s recent post on reducing Haddock’s memory usage for an introduction to the techniques involved.
Some of these improvements included:
Increased sharing of internal identifiers that HLS uses to keep track of different build products and intermediate results.
More compact datastructures to track definition locations for symbols.
Lazily resolving completion information only when a user actually highlights an individual completion item, instead of doing this upfront for all possible completions.
More efficient unloading of dynamically linked objects to improve reloading performance.
As a result of all this work, the heap usage of HLS on Mercury’s codebase went from 12 GB down to 7 GB when starting from scratch (with no disk cache), and from 8 GB down to 3.5 GB when starting with a warm disk cache. Moreover, startup times went from over 4 minutes to around 30 seconds.
Future work
In GHC 9.4, we added support for Multiple Home Units to the GHC API, allowing you to load multiple packages simultaneously into a single GHC API session. Recently, Matthew worked on Cabal to allow loading multiple components into a single GHCi session. We have a work-in-progress PR to allow HLS to exploit this feature so that users can work seamlessly across multiple packages in a single HLS session.
We plan to:
continue investigating and improving HLS memory usage, performance and usability,
help upgrade HLS to newer GHC versions where needed, and
support the volunteer community in promptly providing HLS releases corresponding to GHC releases.
Many thanks to Hasura, Mercury and the donors to the HLS Open Collective for their support in making these improvements possible, and to the whole community of Haskell developers on whose volunteer efforts this work depends.
If you would like to support this work, please consider contributing to the HLS Open Collective. Alternatively, Well-Typed are always interested in projects and looking for funding to improve GHC, HLS, Cabal and other Haskell tools. Please contact info@well-typed.com if we might be able to work with you!
Inscrutable errors such as RTS panics or segfaults, especially when Template Haskell is involved.↩︎