Well-Typed is very happy to announce the first alpha release of the Hackage Security library, along with integration into both
cabal-install and the Hackage server and a tool for managing file-based secure repositories. This release is not yet ready for general use, but we would like to invite interested parties to download and experiment with the library and its integration. We expect a beta release running on the central Hackage server will soon follow.
Hackage Security and related infrastructure is a project funded by the Industrial Haskell Group to secure Hackage, the central Haskell package server. A direct consequence of this work is that we can have untrusted Hackage mirrors (mirror selection is directly supported by the library). A minor but important additional side goal is support for incremental updates of the central Hackage index (only downloading information about new packages, rather than all packages).
TL;DR: Hackage will be more secure, more reliable and faster, and
cabal update should generally finish in seconds.
Security is notoriously difficult to get right, so rather than concocting something ad-hoc the Hackage Security framework is based on TUF, The Update Framework. TUF is a collaboration between security researches at the University of Arizona, the University of Berkeley and the Univerity of Washington, as well as various developers of the Tor project. It is a theory specifically designed for securing software update systems, and suits the needs of Hackage perfectly.
TUF covers both index signing and author signing. Index signing provides the means to verify that something we downloaded from a mirror is the same as what is available from the central server (along with some other security properties), thus making it possible to set up untrusted mirrors. It does not however deal with compromise of the central server. Author signing allows package authors to sign packages, providing a guarantee that the package you download is the one that the author uploaded. These two concerns are largely orthogonal, and the current project only adds support for index signing. Author signing will be the subject of a later project.
Very briefly, here is how it works. The bits in red refer to new features added as part of the Hackage Security work.
Hackage provides a file 00-index.tar.gz (known as “the index”) which contains the
.cabalfiles for all versions of all packages on the server. It is this file that
cabal updatedownloads, and it is this file that
cabal installuses to find out which packages are available and what their dependencies are.Hackage additionally provides a signed file
/snapshot.json(“the snapshot”), containing a hash of the index. When
cabaldownloads the index it computes its hash and verifies it against the hash recorded in the snapshot. Since mirrors do not have the key necessary to sign the snapshot (“ the snapshot key”), if the hash matches we know that the index we downloaded, and hence all files within, was the same as on the central server.
cabal installone or more packages, the index provides
cabalwith a list of packages and their dependencies. However, the index does not contain the packages themselves.The index does however contain
package.jsonfiles for each version of each package, containing a hash for the
.tar.gzof that package version. Since these files live in the index they are automatically protected by the snapshot key. When
cabaldownloads a package it computes the hash of the package and compares it against the hash recorded in the index; if it matches, we are guaranteed that the package is the same as the package on the central server, as the central server is the only one with access to the snapshot key.
- The client does not have built-in knowledge of the snapshot key. Instead, it can download
/root.json(“the root metadata”) from the server, which contains the public snapshot key. The root metadata itself is signed by the root keys, of which the client does have built-in knowledge. The private root keys must be kept very securely (e.g. encrypted and offline).
This description leaves out lots of details, but the purpose of this blog post is not to give a full overview of TUF. See the initial announcement or the website of The Update Framework for more details on TUF; the Hackage Security project README provides a very detailed discussion on how our implemention of TUF relates to the official TUF specification.
Most of the functionality is provided through a new library called
hackage-security, available from github, designed to be used by clients and servers alike.
Although we have integrated it in
hackage-security library is expressly designed to be useable by different clients as well. For example, it generalizes over the library to use for HTTP requests;
cabal uses hackage-security-HTTP, a thin layer around the HTTP library. However, if a client such as stack wants to use the
hackage-security library to talk to Hackage it may prefer to use hackage-security-http-client instead, a thin layer around the http-client library.
Using the library is very simple. After importing Hackage.Security.Client three functions become available, corresponding to points 1, 2 and 3 above:
checkForUpdates :: Repository -> CheckExpiry -> IO HasUpdates downloadPackage :: Repository -> PackageId -> (TempPath -> IO a) -> IO a bootstrap :: Repository -> [KeyId] -> KeyThreshold -> IO ()
Repositoryis an object describing a (local or remote) repository.
CheckExpiryargument describes whether we should check expiry dates on metadata. Expiry dates are important to prevent attacks where a malicious mirror provides outdated data (see A Look In the Mirror: Attacks on Package Managers, Section 3, Threat Model) but we may occassionally want to accept expired data (for instance, when the central server is down for an extended period of time).
bootstraprepresent the client’s “built-in” knowledge of the root keys we alluded to above. In the case of
cabal-installthese come from the cabal
configfile, which may contain a section such as
remote-repo hackage.haskell.org url: http://hackage.haskell.org/ secure: True root-keys: 2ae741f4c4a5f70ed6e6c48762e0d7a493d8dd265e9cbc6c4037dfc7ceaec70e 32d3db5b4403935c0baf52a2bcb05031784a971ee2d43587288776f2e90609db eed36d2bb15f94628221cde558e99c4e1ad36fd243fe3748e1ee7ad00eb9d628 key-threshold: 2
(this syntax for specifying repositories in the cabal
We have written an example client that demonstrates how to use this API; the example client supports both local and remote repositories and can use
curl, and yet is only just over 100 lines of code.
It is important to realize that servers need not be running the Hackage software; mirrors of the central Hackage server may (and typically will) be simple HTTP file servers, and indeed company-internal package servers may choose not to use the Hackage software at all, using only file servers. We provide a hackage-security utility for managing such file-based repositories; see below for details.
Trying it out
There are various ways in which you can try out this alpha release, depending on what precisely you are interested in.
Resources at a glance
hackage-security library github tag “pre-release-alpha” cabal-install github branch “using-hackage-security” hackage github branch “using-hackage-security”
Secure Hackage snapshots
We provide two almost-complete secure (but mostly static) Hackage snapshots, located at
If you want to use
cabal to talk to these repositories, you will need to download and build it from the using-hackage-security branch. Then change your
config and add a new section:
remote-repo alpha-snapshot url: http://hackage.haskell.org/security-alpha/mirror1 secure: True root-keys: 89e692e45b53b575f79a02f82fe47359b0b577dec23b45d846d6e734ffcc887a dc4b6619e8ea2a0b72cad89a3803382f6acc8beda758be51660b5ce7c15e535b 1035a452fd3ada87956f9e77595cfd5e41446781d7ba9ff9e58b94488ac0bad7 key-threshold: 2
It suffices to point
cabal to either mirror; TUF and the
hackage-security library provide built-in support for providing clients with a list of mirrors. During the first check for updates
cabal will download this list, and then use either mirror thereafter. Note that if you wish you can set the
key-threshold to 0 and not specify any root keys; if you do this, the initial download of the
root information will not be verified, but all access will be secure after that.
These mirrors are almost complete, because the first mirror has an intentional problem: the latest version of
generics-sop does not match its signed hash (simulating an evil attempt from an attacker to replace the
generics-sop library with
DoublyEvil-0.3.142). If you attempt to
cabal get this library
cabal should notice something is amiss on this mirror, and automatically try again from the second mirror (which has not been “compromised”):
# cabal get generics-sop Downloading generics-sop-0.1.1.2... Selected mirror http://hackage.haskell.org/security-alpha/mirror1 Downloading package generics-sop-0.1.1.2 Exception Invalid hash for .../generics-sop-0.1.1.2.tar45887.gz when using mirror http://hackage.haskell.org/security-alpha/mirror1 Selected mirror http://hackage.haskell.org/security-alpha/mirror2 Downloading package generics-sop-0.1.1.2 Unpacking to generics-sop-0.1.1.2/
(It is also possible to use the example client to talk to these mirrors, or indeed to a secure repo of your own.)
Setting up a secure file-based repo
If you want to experiment with setting up your own secure repository, the easiest way to do this is to set up a file based repository using the hackage-security utility. A file based repository (as opposed to one running the actual Hackage software) is much easier to set up and will suffice for many purposes.
- Create a directory
~/my-secure-repocontaining a single subdirectory
~/my-secure-repo/package. Put whatever packages you want to make available from your repo in this subdirectory. At this point your repository might look like
``` ~/my-secure-repo/package/basic-sop-0.1.0.5.tar.gz ~/my-secure-repo/package/generics-sop-0.1.1.1.tar.gz ~/my-secure-repo/package/generics-sop-0.1.1.2.tar.gz ~/my-secure-repo/package/json-sop-0.1.0.4.tar.gz ~/my-secure-repo/package/lens-sop-0.1.0.2.tar.gz ~/my-secure-repo/package/pretty-sop-0.1.0.1.tar.gz ``` (because obviously the [generics-sop](http://hackage.haskell.org/package/generics-sop) packages are the first things to come to mind when thinking about which packages are important to secure.) Note the flat directory structure: different packages and different versions of those packages all live in the one directory.
Create public and private keys:
# hackage-security create-keys --keys ~/my-private-keys
This will create a directory structure such as
~/my-private-keys/mirrors/id01.private ~/my-private-keys/mirrors/.. ~/my-private-keys/root/id04.private ~/my-private-keys/root/.. ~/my-private-keys/snapshot/id07.private ~/my-private-keys/target/id08.private ~/my-private-keys/target/.. ~/my-private-keys/timestamp/id11.private
containing keys for all the various TUF roles (proper key management is not part of this alpha release).
Note that these keys are stored outside of the repository proper.
Create the initial TUF metadata and construct an index using
# hackage-security bootstrap \ --repo ~/my-secure-repo \ --keys ~/my-private-keys
This will create a directory
.cabalfiles (extracted from the package tarballs) and TUF metadata for all packages
~/my-secure-repo/index/basic-sop/0.1.0.5/basic-sop.cabal ~/my-secure-repo/index/basic-sop/0.1.0.5/package.json ~/my-secure-repo/index/generics-sop/0.1.1.1/generics-sop.cabal ~/my-secure-repo/index/generics-sop/0.1.1.1/package.json ...
and package the contents of that directory up as the index tarball
~/my-secure-repo/00-index.tar.gz; it will also create the top-level metadata files
~/my-secure-repo/mirrors.json ~/my-secure-repo/root.json ~/my-secure-repo/snapshot.json ~/my-secure-repo/timestamp.json
The timestamp and snapshot are valid for three days, so you will need to resign these files regularly using
# hackage-security update \ --repo ~/my-secure-repo \ --keys ~/my-private-keys
You can use the same command whenever you add any additional packages to your repository.
If you now make this directory available (for instance, by pointing Apache at it) you should be able to use
cabalto access it, in the same way as described above for accessing the secure Hackage snapshots. You can either set
key-thresholdto 0, or else copy in the root key IDs from the generated
Setting up a secure Hackage server
If you are feeling adventurous you can also try to set up your own secure Hackage server. You will need to build Hackage from the using-secure-hackage branch.
You will need to create a subdirectory
TUF inside Hackage’s
datafiles/ directory, containing 4 files:
datafiles/TUF/mirrors.json datafiles/TUF/root.json datafiles/TUF/snapshot.private datafiles/TUF/timestamp.private
containing the list of mirrors, the root metadata, and the private snapshot and timestamp keys. You can create these files using the
- Use the
create-keysas described above to create a directory with keys for all roles, and then copy over the snapshot and timestamp keys to the
- Use the
create-mirrorscommands to create the root and mirrors metadata. The
create-mirrorsaccepts an arbitrary list of mirrors to be added to the mirrors metadata, should you wish to do so.
Note that the
mirrors.json files are served as-is by Hackage, they are not used internally; the snapshot and timestamp keys are of course used to sign the snapshot and the timestamp.
Once you have created and added these files, everything else should Just Work(™). When you start up your server it will create TUF metadata for any existing packages you may have (if you are migrating an existing database). It will create a snapshot and a timestamp file; create metadata for any new packages you upload and update the snapshot and timestamp; and resign the snapshot and timestamp nightly. You can talk to the repository using
cabal as above.
If you a have Hackage server containing a lot of packages (a full mirror of the central Hackage server, for instance) then migration will be slow; it takes approximately an hour to compute hashes for all packages on Hackage. If this would lead to unacceptable downtime you can use the precompute-fileinfo tool to precompute hashes for all packages, given a recent backup. Copy the file created by this tool to
datafiles/TUF/md5-to-sha256.map before doing the migration. If all hashes are precomputed migration only takes a few minutes for a full Hackage snapshot.
If you want to experiment with integrating the
hackage-security library into your own software, the example client is probably the best starting point for integration in client software, and the hackage-security utility is probably a good starting point for integration in server software.
Please report any bugs or comments you may have as GitHub issues.
This is an alpha release, intended for testing by people with a close interest in the Hackage Security work. The issue tracker contains a list of issues to be resolved before the beta release, at which point we will make the security features available on the central Hackage server and make a patched
cabal available in a more convenient manner. Note that the changes to Hackage are entirely backwards compatible, so existing clients will not be affected.