grapesy: industrial strength gRPC library for Haskell

Well-Typed are delighted to announce the release of grapesy (Hackage, GitHub), an industial strength Haskell library providing support for gRPC, a modern open source high performance Remote Procedure Call (RPC) framework developed by Google. The library has the following features:

• Support for both gRPC clients and gRPC servers.

• Full compliance with the core gRPC specification, passing all official interoperability tests.

• Parametric in the choice of message format; Protobuf is the most common choice for gRPC and is of course supported, as is JSON¹. There is also support for general binary (“raw”) messages, and adding additional formats is easy and can be done without modifying grapesy itself.

• Client-side and server-side support for the Protobuf common communication patterns: non-streaming, client-side streaming, server-side streaming, and bidirectional streaming. Use of these patterns is independent of the choice of message encoding, and is strictly optional.

• For the specific case of Protobuf, support for Protobuf rich errors.

• Support for metadata: request initial metadata, response initial metadata, and response trailing metadata.

• Support for all the common gRPC compression algorithms: gzip and zlib (both through the zlib package), as well as snappy (through a new package snappy-c, developed for this purpose). Bespoke compression algorithms can also be used, and compression can be disabled on a per-message basis.

• Support for both unencrypted and encrypted connections (TLS).

• Support for cancellation, both through deadlines/timeouts (server-side cancellation) as well as through terminating a RPC early (client-side cancellation).²

• Flow control: we are careful to use back-pressure to limit traffic, ultimately relying on HTTP2 flow control (which can be adjusted through the HTTP2Settings, primarily the stream window size and the connection window size).

• Support for Wait-for-Ready, where the connection to a server can be (re)established in the background, rather than an RPC failing if the server is not immediately available. Note that this must be enabled explicitly (as per the spec).

• Asynchronous design: operations happen in the background whenever possible (opening a connection, initiating an RPC, sending a message), and exceptions are only raised when directly interacting with those background processes. For example, when a client disconnects from the server, the corresponding handler will only get an exception if it attempts any further communication with that client. This is particularly important in RPC servers, which may need to complete certain operations even if the client that requested those operations did not stick around to wait for them.

• Type safety: the types of inputs (messages sent from the client to the server) and outputs (messages from the server to the client), as well as the types of the request and response metadata, are all determined from the choice of a specific RPC. In addition, for Protobuf servers we can guarantee at the type-level that all methods are handled (or explicitly declared as unsupported).

• Extensive documentation: this blog post contains a number of tutorials that highlight the various parts of grapesy, and the Haddock documentation is comprehensive.

The library is designed to be robust:

• Exception safety: all exceptions, in both client and in server contexts, are caught and handled in context appropriate ways; they are never simply “lost”. Server-side exceptions are reported as gRPC errors on the client; handlers can also throw any of the standard gRPC errors.

• Deals correctly with broken deployments (clients or servers that do not conform to the gRPC specification). This includes things such as dealing with non-200 HTTP status codes, correctly responding to unsupported content types (for which the gRPC spec mandates a different resolution on servers and clients), dealing with servers that don’t respect timeouts, etc.

• Native Haskell library (does not bind to any C or C++ libraries).

• Comes with a comprehensive test suite, which has been instrumental in achieving high reliability, as well as finding problems elsewhere in the network stack; as part of the development of grapesy we have also made numerous improvements to http2 and related libraries³. Many thanks to Kazu Yamamoto for being so receptive to all our PRs and willing to discuss all the issues we found, as well as his hard work on these core infrastructure libraries!

• No memory leaks: even under stress conditions, memory usage is completely flat in both the client and the server.

• Good performance, on par with the official Java implementation.

Developing a library of this nature is a significant investment, and so Well-Typed is thankful to Anduril for sponsoring the work.

Quickstart

In this section we explain how to get started, in the style of the official Quickstart guide. You can also use the Quickstart example as a basic template for your own gRPC applications.

gRPC tools

Neither gRPC nor grapesy requires the use of Protobuf, but it is the most common way of using gRPC, and it is used by both the Quickstart tutorial as well as the Basics tutorial. You will therefore need to install the protobuf buffer compiler protoc, which can usually be done using your system’s package manager; see Protobuf Buffer Compiler Installation for details.

Download the example

If you want to work through this quick start, you will need to clone the grapesy repository:

$ git clone https://github.com/well-typed/grapesy.git
$ cd grapesy/tutorials/quickstart

Run a gRPC application

From the grapesy/tutorials/quickstart directory, run the server

$ cabal run greeter_server

From another terminal, run the client:

$ cabal run greeter_client

If all went well, you should see the server responding to the client with

Proto {message: "Hello, you"}

Update the gRPC service

Now let’s try to add another method to the Greeter service. This service is defined using protocol buffers; for an introduction to gRPC in general and Protobuf specifically, you may wish to read the official Introduction to gRPC; we will also see more examples of Protobuf below in the Basics tutorial. You can find the definition for the quickstart tutorial in tutorials/quickstart/proto/helloworld.proto:

syntax = "proto3";

// The greeting service definition.
service Greeter {
  // Sends a greeting
  rpc SayHello (HelloRequest) returns (HelloReply) {}
}

// The request message containing the user's name.
message HelloRequest {
  string name = 1;
}

// The response message containing the greetings
message HelloReply {
  string message = 1;
}

Let’s add another method to this service, with the same request and response types:

service Greeter {
  // Sends a greeting
  rpc SayHello (HelloRequest) returns (HelloReply) {}

  // Sends another greeting
  rpc SayHelloAgain (HelloRequest) returns (HelloReply) {}
}

Generate gRPC code

The example is set up to use a custom Cabal setup script to automatically compile the proto definitions; see proto-lens-setup for a detailed discussion on how to do this. If you prefer not to use custom setup scripts in your own projects, it is also possible to run the Protobuf compiler manually; see section Manually running the protocol compiler of the proto-lens-protoc documentation.

This means that to re-run the Protobuf compiler, it suffices to build either the client or the server; let’s attempt to build the server:

$ cabal build greeter_server

You should see a type error:

app/Server.hs:13:7: error: [GHC-83865]
    • Couldn't match type: '[]
                     with: '[Protobuf Greeter "sayHelloAgain"]

This is telling you that the server is incomplete: we are missing a handler for the new sayHelloAgain method.

Update the server

To update the server, edit Server.hs and add:

sayHelloAgain :: Proto HelloRequest -> IO (Proto HelloReply)
sayHelloAgain req = do
    let resp = defMessage & #message .~ "Hello again, " <> req ^. #name
    return resp

Then update methods to list the new handler:

methods :: Methods IO (ProtobufMethodsOf Greeter)
methods =
      Method (mkNonStreaming sayHello)
    $ Method (mkNonStreaming sayHelloAgain)
    $ NoMoreMethods

Update the client

Unlike the server, the change to the service definition does not require changes to the client. The server must implement the new method, but the client does not have to call it. Of course, it is more interesting when it does, so let’s add another call to Client.hs:

withConnection def server $ \conn -> do
  let req = defMessage & #name .~ "you"
  resp <- nonStreaming conn (rpc @(Protobuf Greeter "sayHello")) req
  print resp
  resp2 <- nonStreaming conn (rpc @(Protobuf Greeter "sayHelloAgain")) req
  print resp2

Run

After restarting greeter_server, running greeter_client should now output

Proto {message: "Hello, you"}
Proto {message: "Hello again, you"}

Basics

In this section we delve a little deeper, following the official Basics tutorial, which introduces the RouteGuide service. From the official docs:

Our example is a simple route mapping application that lets clients get information about features on their route, create a summary of their route, and exchange route information such as traffic updates with the server and other clients.

You can find the example in the tutorials/basics directory of the grapesy repo.

Defining the service

The RouteGuide example illustrates the four different kinds of communication patterns that Protobuf services can have. You can find the full service definition in tutorials/basics/proto/route_guide.proto:

• Non-streaming: client sends a single input, server replies with a single output:

  // Obtains the feature at a given position.
  rpc GetFeature(Point) returns (Feature) {}

• Server-side streaming: client sends a single input, server can respond with any number of outputs:

  // Obtains the Features available within the given Rectangle.
  rpc ListFeatures(Rectangle) returns (stream Feature) {}

• Client-side streaming: client can send any number of inputs, after which the server responds with a single output:

  // Accepts a stream of Points on a route being traversed, returning a
  // RouteSummary when traversal is completed.
  rpc RecordRoute(stream Point) returns (RouteSummary) {}

• Bidirectional streaming: the client and the server can exchange messages at will:

  // Accepts a stream of RouteNotes sent while a route is being traversed,
  // while receiving other RouteNotes (e.g. from other users).
  rpc RouteChat(stream RouteNote) returns (stream RouteNote) {}

There is explicit support in grapesy for these four communication patterns, both for defining servers and for defining clients. In addition, there is a lower-level API which provides more control over the communication; we will see some examples in Beyond the basics.

Generated code

As in the Quickstart, we have set things up in the example to automatically generate Haskell code from the .proto definition. There is however one more thing that we need to take care of, which we glossed over previously. The .proto definition is sufficient to determine the types of the methods of the service, their arguments, and their results. But it does not say anything about the type of any metadata. We don’t need any metadata in this example, so we can declare the following module:

module Proto.API.RouteGuide (
    module Proto.RouteGuide
  ) where

import Network.GRPC.Common
import Network.GRPC.Common.Protobuf

import Proto.RouteGuide

type instance RequestMetadata          (Protobuf RouteGuide meth) = NoMetadata
type instance ResponseInitialMetadata  (Protobuf RouteGuide meth) = NoMetadata
type instance ResponseTrailingMetadata (Protobuf RouteGuide meth) = NoMetadata

This re-exports module Proto.RouteGuide (which was generated), along with three type family instances that indicate that none of the methods of the RouteGuide require metadata. We will see an example of using metadata later.

Proto wrapper

In the repository you will find an implementation of the logic of the RouteGuide example as a collection of pure functions; see tutorials/basics/src/RouteGuide.hs. For example, the type of the function that looks up which feature exists at a particular point, given the example database of features, is given by:

featureAt :: DB -> Proto Point -> Maybe (Proto Feature)

The precise implementation is not very important for our purposes here, but we should discuss that Proto wrapper. This is a type-level marker that explicitly identifies Protobuf values. Such values don’t behave like regular Haskell values; for example, record fields always have defaults, enums might have unknown values, etc. The idiomatic way of accessing fields of a Proto value is using a lens access and an (overloaded) label; for example, the following expression extracts a field #location from a feature (f :: Proto Feature):

f ^. #location

To construct a Proto value you first create an empty value using defMessage, and then update individual fields with a lens update. For example, here is how we might construct a Proto RouteSummary:

defMessage
  & #pointCount   .~ ..
  & #featureCount .~ ..
  & #distance     .~ ..
  & #elapsedTime  .~ ..

Everything required to work with Protobuf values is (re-)exported from Network.GRPC.Common.Protobuf. In addition, Network.GRPC.Common.Protobuf.Any provides functionality for working with the Protobuf Any type.

Implementing the server

We can use the type checker to help us in the development of the server. We know that we want to implement the methods of the RouteGuide service; if we define

methods :: DB -> Methods IO (ProtobufMethodsOf RouteGuide)
methods db = _

the type checker will tell us that it expects something of this type⁴:

_ :: Methods IO [
    Protobuf RouteGuide "getFeature"
  , Protobuf RouteGuide "listFeatures"
  , Protobuf RouteGuide "recordRoute"
  , Protobuf RouteGuide "routeChat"
  ]

We can therefore refine methods to

methods :: DB -> Methods IO (ProtobufMethodsOf RouteGuide)
methods db =
      Method _getFeature
    $ Method _listFeatures
    $ Method _recordRoute
    $ Method _routeChat
    $ NoMoreMethods

at which point the type checker informs us:

_getFeature   :: ServerHandler' NonStreaming    IO (Protobuf RouteGuide "getFeature")
_listFeatures :: ServerHandler' ServerStreaming IO (Protobuf RouteGuide "listFeatures")
_recordRoute  :: ServerHandler' ClientStreaming IO (Protobuf RouteGuide "recordRoute")
_routeChat    :: ServerHandler' BiDiStreaming   IO (Protobuf RouteGuide "routeChat")

We can therefore refine once more to

methods :: DB -> Methods IO (ProtobufMethodsOf RouteGuide)
methods db =
      Method (mkNonStreaming    $ _getFeature)
    $ Method (mkServerStreaming $ _listFeatures)
    $ Method (mkClientStreaming $ _recordRoute)
    $ Method (mkBiDiStreaming   $ _routeChat)
    $ NoMoreMethods

The resulting types will depend on the communication pattern (non-streaming, client-side streaming, etc.). We will discuss them one by one.

Non-streaming RPC

The first method is a non-streaming RPC, for which the type checker infers:

_getFeature :: Proto Point -> IO (Proto Feature)

That is, we are given a point of interest, and must return “the” feature at that point. We will also need the database of features. The implementation is straight-forward, and essentially just wraps the pure function featureAt:

getFeature :: DB -> Proto Point -> IO (Proto Feature)
getFeature db p = return $ fromMaybe (defMessage & #location .~ p) (featureAt db p)

The only minor complication here is that we need to construct some kind of default location for when there is no feature found at point p.

Server-side streaming

For server-side streaming we are given the input from the client, along with a function that we can use to send outputs back to the client:

_listFeatures :: Proto Rectangle -> (NextElem (Proto Feature) -> IO ()) -> IO ()

NextElem is similar to Maybe:

data NextElem a = NoNextElem | NextElem !a

but with a more specialized API. For example, it offers

forM_ :: Monad m => [a] -> (NextElem a -> m ()) -> m ()

which will invoke the specified callback NextElem x for all x in the list, and then invoke the callback once more with NoNextElem. We can use this to implement listFeatures:

listFeatures :: DB -> Proto Rectangle -> (NextElem (Proto Feature) -> IO ()) -> IO ()
listFeatures db r send = NextElem.forM_ (featuresIn db r) send

Client-side streaming

For client-side streaming we are given a function to receive inputs from the client, and must produce a single output to be sent back to the client:

_recordRoute :: IO (NextElem (Proto Point)) -> IO (Proto RouteSummary)

To implement it, we can use another function from the NextElem API:

collect :: Monad m => m (NextElem a) -> m [a]

The only other complication is that the function which constructs the RouteSummary also wants to know how long it took to collect all points:

recordRoute :: DB -> IO (NextElem (Proto Point)) -> IO (Proto RouteSummary)
recordRoute db recv = do
    start <- getCurrentTime
    ps    <- NextElem.collect recv
    stop  <- getCurrentTime
    return $ summary db (stop `diffUTCTime` start) ps

Bidirectional streaming

For bidirectional streaming finally we get two functions: one to receive inputs from the client, and one to send outputs back to the client:

_routeChat ::
     IO (NextElem (Proto RouteNote))
  -> (NextElem (Proto RouteNote) -> IO ())
  -> IO ()

The implementation is straight-forward and does not require any new grapesy features; you can find it in tutorials/basics/app/Server.hs.

Top-level application

The main server application then looks like this:

main :: IO ()
main = do
    db <- getDB
    runServerWithHandlers def config $ fromMethods (methods db)
  where
    config :: ServerConfig
    config = ServerConfig {
          serverInsecure = Just (InsecureConfig Nothing defaultInsecurePort)
        , serverSecure   = Nothing
        }

The first parameter to runServerWithHandlers are the server parameters. The most important parameters to consider are serverTopLevel and serverExceptionToClient. These two are related, and describe how to deal with exceptions:

• serverTopLevel says what to do with exceptions server-side; by default it simply prints them to stderr
• serverExceptionToClient says what information to include in the error sent to the client; by default it calls displayException. You may wish to override this if you are concerned about leaking security sensitive information.

Implementing the client

You can find the complete client in tutorials/basics/app/Client.hs.

Connecting to the server

Before we can make any RPCs, we have to connect to the server:

main :: IO ()
main =
    withConnection def server $ \conn -> do
      ..
  where
    server :: Server
    server = ServerInsecure $ Address "127.0.0.1" defaultInsecurePort Nothing

The first argument are the connection parameters, the most important of which is probably the reconnection policy which (amongst other things) is used to enable Wait-for-Ready semantics.

Simple RPC

We already saw how to make a simple non-streaming RPC in the quickstart:

getFeature :: Connection -> IO ()
getFeature conn = do
    let req = defMessage
                & #latitude  .~  409146138
                & #longitude .~ -746188906
    resp <- nonStreaming conn (rpc @(Protobuf RouteGuide "getFeature")) req
    print resp

We construct a request, do the RPC, and print the response.

Server-side streaming

When we make a server-side streaming RPC, we are given a function we can call to get all of the server outputs:

listFeatures :: Connection -> IO ()
listFeatures conn = do
    let req = ..
    serverStreaming conn (rpc @(Protobuf RouteGuide "listFeatures")) req $ \recv ->
      NextElem.whileNext_ recv print

Here we are using another function from the NextElem API, in a sense dual to the one we used server-side; for comparison, both types:

forM_      :: Monad m => [a] -> (NextElem a -> m ()) -> m ()
whileNext_ :: Monad m => m (NextElem a) -> (a -> m b) -> m ()

Client-side streaming

To make a client-side streaming RPC, we are given a function that we can use to send inputs to the server; once we are done sending all inputs, we then receive the final (and only) output from the server:

recordRoute :: Connection -> IO ()
recordRoute conn = do
    resp <- clientStreaming_ conn (rpc @(Protobuf RouteGuide "recordRoute")) $ \send -> do
      replicateM_ 10 $ do
        let p = (db !! i) ^. #location
        send $ NextElem p
        threadDelay 500_000 -- 0.5 seconds
      send NoNextElem
    print resp

Bidirectional streaming

Finally, for bidirectional streaming we are given two functions, one to send, and one to receive. In this particular case, we can first send all inputs and then receive all outputs, but in general these can be interleaved in any order:

routeChat :: Connection -> IO ()
routeChat conn = do
    biDiStreaming conn (rpc @(Protobuf RouteGuide "routeChat")) $ \send recv -> do
      NextElem.forM_ messages send
      NextElem.whileNext_ recv print
  where
    messages = ..

See also The Haskell Unfolder, episode 27: duality for a more in-depth look into the duality between these various communication patterns.

End of output

When we discussed the client-side implementation of a client-side streaming RPC, we used function clientStreaming_:

clientStreaming_ ::
     ..
  -> (    (NextElem (Input rpc) -> m ())
       -> m ()
     )
  -> m (Output rpc)

The callback is given a function (which we called send) to send outputs to the server. The problem with this approach is that it’s possible to forget to call this function; in particular, it’s quite easy to forget the final

send NoNextElem

to indicate to the server that there is no further input coming. In some cases iteration functions such as NextElem.forM_ can take care of this, but this could also result in the opposite problem, calling send on a NextElem after NoNextElem has already been sent.

In short: make sure to send NoNextElem in clients or servers that stream values to their peer:

• If you forget to do this in a server handler, grapesy will assume this is a bug and throw a HandlerTerminated exception, which will be reported as a gRPC exception with an unknown error on the client.

• If you forget to do this in a client, grapesy will assume that you intend to cancel the RPC. The server will see call closed suddenly⁵, and on the client this will result in a gRPC exception with “cancelled” as the error.

Sending more elements after sending NoNextElem will result in SendAfterFinal exception.

Side note. In principle it is possible to give clientStreaming_ a different type:

-- Alternative definition, not actually used in grapesy
clientStreaming_ ::
     ..
  -> m (NextElem (Input rpc))
  -> m (Output rpc)

In this style there is no callback at all; instead, we must provide an action that produces the next element one by one, and the library will ensure that the function is called repeatedly until it returns NoNextElem. This amounts to inversion of control: you don’t call a function to send each value, but the library calls you to ask what the next value to be sent is. This provides stronger guarantees that the communication pattern is implemented correctly, but we deemed the cost too high: it results in quite an awkward programming model. Of course, if you want to, nothing stops you from defining such an API on top of the API offered by grapesy.

Beyond the basics

In this section we describe some of the more advanced features of grapesy.

Using the low-level API

Both the Quickstart and the Basics tutorial used the StreamType API, which captures the four different communication patterns (aka streaming types) used in Protobuf, both on the server and on the client: non-streaming, server-side streaming, client-side streaming, and bidirectional streaming. Although these four communication patterns originate in Protobuf, in grapesy they are not Protobuf specific and can also be used with other message encodings.

The high-level API will probably suffice for the vast majority of gRPC applications, but not quite all, and grapesy also offers a low-level API. The most important reasons to use the low-level API instead are:

• Making sure that the final message is marked as final; we discuss this in more detail in this section in Final elements.

• Sending and receiving metadata; we will discuss this in detail in the next section Using metadata.

• Preference: some people may simpler prefer the style of the low-level API over the high-level API.

Although the use of the low-level API does come with some responsibilities that are taken care of for you in the high-level API, it is not significantly more difficult to use.

Final elements

When we discussed the high-level API, we saw the NextElem type. The low-level API uses StreamElem instead; here they are side by side:

data NextElem     a = NoNextElem     | NextElem   !a
data StreamElem b a = NoMoreElems !b | StreamElem !a | FinalElem !a !b

There are two differences here:

• When there are no more elements, we record an additional value. This is the metadata to be sent or received after the final element. We will see an example of this below; for RouteGuide this metadata will always be NoMetadata, which is a trivial type isomorphic to ():

  data NoMetadata = NoMetadata

• The final element can be marked as final, rather than requiring a separate NoMoreElems value. This may feel like an insignificant difference, but although it is a technicality, in some cases it’s an important technicality.

To understand the need for marking the final element, we need to understand that gRPC messages are transferred over HTTP2 DATA frames. It’s not necessarily true that one frame corresponds to one message, but let’s for the sake of simplicity assume that it is. Then in order to send 3 messages, we have two options:

corresponding to

    [StreamElem 1, StreamElem 2, StreamElem 3, NoMoreElems NoMetadata]
and [StreamElem 1, StreamElem 2, FinalElem 3 NoMetadata]

respectively. This matters because some servers report an error if they receive a message that they expect will be the final message, but the corresponding HTTP2 DATA frame is not marked END_STREAM. This is not completely unreasonable: after all, waiting to receive the next DATA frame might be a blocking operation.

This is particularly important in cases where a server (or client) only expects a single message (non-streaming, client-side streaming, expecting a single output from the server, or server-side streaming, expecting a single input from the client). It is much less critical in other situations, which is why the high-level API gets away with using NextElem instead of StreamElem (which it uses only when multiple messages are expected).

On the server

To use the low-level API on the server, you can either use RawMethod to use the low-level API for some (or all) of the methods of an API, or you avoid the use of fromMethods altogether. The latter option is primarily useful if you don’t have a type-level description of your service available. If you do, the first option is safer:

methods :: DB -> Methods IO (ProtobufMethodsOf RouteGuide)
methods db =
      RawMethod (mkRpcHandler $ getFeature   db)
    $ RawMethod (mkRpcHandler $ listFeatures db)
    $ RawMethod (mkRpcHandler $ recordRoute  db)
    $ RawMethod (mkRpcHandler $ routeChat      )
    $ NoMoreMethods

It is also possible to use the high-level API for most methods, and escape to the low-level API for those methods that need it.

Unlike with the high-level API, the signature of all handlers that use the low-level API is the same:

getFeature   :: DB -> Call (Protobuf RouteGuide "getFeature")   -> IO ()
listFeatures :: DB -> Call (Protobuf RouteGuide "listFeatures") -> IO ()
recordRoute  :: DB -> Call (Protobuf RouteGuide "recordRoute")  -> IO ()
routeChat    ::       Call (Protobuf RouteGuide "routeChat")    -> IO ()

The most important two functions⁶ for communication on the server are recvInput and sendOutput:

recvInput  :: Call rpc -> IO (StreamElem NoMetadata (Input rpc))
sendOutput :: Call rpc -> StreamElem (ResponseTrailingMetadata rpc) (Output rpc) -> IO ()

For convenience there are also some derived functions available; for example, here is getFeature again, now using the low-level API:

getFeature :: DB -> Call (Protobuf RouteGuide "getFeature") -> IO ()
getFeature db call = do
    p <- recvFinalInput call
    sendFinalOutput call (
        fromMaybe (defMessage & #location .~ p) (featureAt db p)
      , NoMetadata
      )

The StreamElem API also offers some iteration functions similar to the ones offered by NextElem; for example, here is listFeatures:

listFeatures :: DB -> Call (Protobuf RouteGuide "listFeatures") -> IO ()
listFeatures db call = do
    r <- recvFinalInput call
    StreamElem.forM_ (featuresIn db r) NoMetadata (sendOutput call)

The full server definition is available in tutorials/lowlevel/app/Server.hs.

On the client

The main function to make an RPC using the low-level API is withRPC. For example, here is getFeature:

getFeature :: Connection -> IO ()
getFeature conn = do
    let req = ..
    withRPC conn def (Proxy @(Protobuf RouteGuide "getFeature")) $ \call -> do
      sendFinalInput call req
      resp <- recvFinalOutput call
      print resp

The second argument to withRPC are the call parameters, of which there are two important ones: the timeout for this RPC, and the request metadata. (When using the high-level API the only way to set a timeout is to specify the default RPC timeout for the connection.)

End of output, revisited

At the end of the basics tutorial, we emphasized the importance of indicating end of output for streaming clients and server handlers. The discussion there is relevant when using the low-level API as well, with some additional caveats:

• In the high-level API, the library can take care of marking the (only) value for non-streaming output as final; in the low-level API, this is your own responsibility, either through calling sendFinalInput / sendFinalOutput or through calling sendInput / sendOutput and constructing the StreamElem manually.

• For streaming outputs, you can use sendEndOfInput (clients) or sendTrailers (servers) to indicate end of output after the fact (like NoNextElem does), or use sendFinalInput / sendFinalOutput to mark the final element as final when you send it. This should be preferred whenever possible.

Using metadata

As an example of using metadata, let’s construct a simple file server which tells the client the size of the file to be downloaded as the initial response metadata, then streams the contents of the file as a series of chunks, and finally reports a SHA256 hash of the file contents in the trailing response metadata. The client can use the initial file size metadata to show a progress bar, and the hash in the trailing metadata to verify that everything went well.

You can find the full example in tutorials/metadata.

Service definition

The .proto file is straight-forward:

syntax = "proto3";

package fileserver;

service Fileserver {
  rpc Download (File) returns (stream Partial) {}
}

message File {
  string name = 1;
}

message Partial {
  bytes chunk = 1;
}

As mentioned above, however, the .proto definition does not tell us the type of the metadata. We need to do this in Haskell:

type instance RequestMetadata          (Protobuf Fileserver "download") = NoMetadata
type instance ResponseInitialMetadata  (Protobuf Fileserver "download") = DownloadStart
type instance ResponseTrailingMetadata (Protobuf Fileserver "download") = DownloadDone

data DownloadStart = DownloadStart {
      downloadSize :: Integer
    }
  deriving stock (Show)

data DownloadDone = DownloadDone {
      downloadHash :: ByteString
    }
  deriving stock (Show)

(In this example we make no use of request metadata; see callRequestMetadata for the main entry point for setting request metadata.)

Serialization

In order for the server to be able to send the metadata to the client, we need to be able serialize it as one (or more, or zero) headers/trailers. This means we must give an instance of BuildMetadata:

instance BuildMetadata DownloadStart where
  buildMetadata DownloadStart{downloadSize} = [
        CustomMetadata "download-size" $ C8.pack (show downloadSize)
      ]

instance BuildMetadata DownloadDone where
  buildMetadata DownloadDone{downloadHash} = [
        CustomMetadata "download-hash-bin" downloadHash
      ]

Note the use of the -bin suffix for the name of the download-hash-bin trailer: this indicates that this is metadata containing binary data, and that it must be Base64-encoded; grapesy will automatically take care of encoding and decoding for binary metadata.

We need to take care of one more thing. The HTTP2 spec mandates that clients must be informed up-front which trailing headers they can expect. In grapesy this comes down to giving an instance of StaticMetadata:

instance StaticMetadata DownloadDone where
  metadataHeaderNames _ = ["download-hash-bin"]

This can be an over-approximation but not an under-approximation; if you return a trailer in BuildMetadata that was not declared in StaticMetadata, then grapesy will throw an exception.

Deserialization

For deserialization we must provide an instance of ParseMetadata, which is given all metadata headers to parse. In our example this is relatively simple because our metadata uses only a single header:

instance ParseMetadata DownloadStart where
  parseMetadata md =
      case md of
        [CustomMetadata "download-size" value]
          | Just downloadSize <- readMaybe (C8.unpack value)
          -> return $ DownloadStart{downloadSize}
        _otherwise
          -> throwM $ UnexpectedMetadata md

instance ParseMetadata DownloadDone where
  parseMetadata md =
      case md of
        [CustomMetadata "download-hash-bin" downloadHash]
          -> return $ DownloadDone{downloadHash}
        _otherwise
          -> throwM $ UnexpectedMetadata md

These particular instances will throw an error if additional metadata is present. This is a choice, and instead we could simply ignore any additional headers. There is no single right answer here: ignoring additional metadata runs the risk of not realizing that the peer is trying to tell you something important, but throwing an error runs the risk of unnecessarily aborting an RPC.

Specifying initial response metadata

The metadata that is sent to the client with the response headers can be overridden with setResponseInitialMetadata. This can be done at any point before initiating the request, either explicitly using initiateResponse or implicitly by sending the first output to the client using sendOutput and related functions.

Most server handlers however don’t care about metadata, and prefer not to have to call to setResponseInitialMetadata at all. For this reason mkRpcHandler has type

mkRpcHandler :: Default (ResponseInitialMetadata rpc) => ..

This constraint is inherited by the high-level API, which doesn’t support metadata at all:

Method :: (Default (ResponseInitialMetadata rpc), Default (ResponseTrailingMetadata rpc)) => ..

Crucially, there is a Default instance for NoMetadata:

instance Default NoMetadata where
  def = NoMetadata

In our case however we cannot provide a Default instance, because the initial metadata depends on the file size. We therefore use mkRpcHandlerNoDefMetadata instead:

methods :: Methods IO (ProtobufMethodsOf Fileserver)
methods =
      RawMethod (mkRpcHandlerNoDefMetadata download)
    $ NoMoreMethods

This means we must call setResponseInitialMetadata in the handler; if we don’t, an exception will be raised when the response is initiated.

Server handler

Since we are using the low-level API (we must, if we want to deal with metadata), the server handler has this signature:

download :: Call (Protobuf Fileserver "download") -> IO ()
download call = do

We wait for the request from the client, get the file size, set the response initial metadata, and initiate the response. Explicitly initiating the response in this manner is not essential, but it means that the file size is sent to the client (along with the rest of the response headers) before the first chunk is sent; in some cases this may be important:

req :: Proto File <- recvFinalInput call
let fp :: FilePath
    fp = Text.unpack (req ^. #name)

fileSize <- getFileSize fp
setResponseInitialMetadata call $ DownloadStart fileSize
initiateResponse call

We then open the file the client requested, and keep reading chunks until we have reached end of file. Although it is probably not particularly critical in this case, we follow the recommendations from End of output, revisited and mark the final chunk as the final output to the client, as opposed to telling the client that no more outputs are available after the fact.

withFile fp ReadMode $ \h -> do
  let loop :: SHA256.Ctx -> IO ()
      loop ctx = do
          chunk <- BS.hGet h defaultChunkSize
          eof   <- hIsEOF h

          let resp :: Proto Partial
              resp = defMessage & #chunk .~ chunk

              ctx' :: SHA256.Ctx
              ctx' = SHA256.update ctx chunk

          if eof then
            sendFinalOutput call (resp, DownloadDone $ SHA256.finalize ctx')
          else do
            sendNextOutput call resp
            loop ctx'

  loop SHA256.init

When we send the final output, we must also include the hash that we computed as we were streaming the file to the client.

Client

Let’s first consider how to process the individual chunks that we get from the server. We do this in an auxiliary function processPartial:

processPartial ::
     Handle
  -> Proto Partial
  -> ProgressT (StateT SHA256.Ctx IO) ()
processPartial h partial = do
    liftIO $ BS.hPut h chunk
    modify $ flip SHA256.update chunk
    updateProgressBar $ BS.length chunk
  where
    chunk :: ByteString
    chunk = partial ^. #chunk

We do three things in this function: write the chunk to disk, update the hash, and update the progress bar; this uses StateT to keep track of the partially computed hash, and ProgressT for a simple progress bar (ProgressT is defined in tutorials/metadata/app/ProgressT.hs; its details are not important here).

This in hand, we can now define the main client function. We are given some file inp that we are interested in downloading, and a path out where we want to store it locally. Like in the server, here too we must use the low-level API, so the client starts like this:

download :: Connection -> String -> String -> IO ()
download conn inp out = do
    withRPC conn def (Proxy @(Protobuf Fileserver "download")) $ \call -> do
      sendFinalInput call $ defMessage & #name .~ Text.pack inp

We then wait for the initial response metadata, telling us how big the file is:

DownloadStart{downloadSize} <- recvResponseInitialMetadata call

We then use StreamElem.whileNext_ again to process all the chunks using processPartial that we already discussed, unwrap the monad stack, and finally do a hash comparison:

(DownloadDone{downloadHash = theirHash}, ourHash) <-
  withFile out WriteMode $ \h ->
    flip runStateT SHA256.init . runProgressT downloadSize $
      StreamElem.whileNext_ (recvOutput call) (processPartial h)

putStrLn $ "Hash match: " ++ show (theirHash == SHA256.finalize ourHash)

Custom monad stack

In this section we will briefly discuss how to use custom monad stacks. You can find the full tutorial in tutorials/monadstack; it is a variant on Basics tutorial.

On the server

Most of the server handlers in for the RouteGuide service need to take the DB as an argument:

getFeature   :: DB -> Proto Point -> IO (Proto Feature)
listFeatures :: DB -> Proto Rectangle -> (NextElem (Proto Feature) -> IO ()) -> IO ()
recordRoute  :: DB -> IO (NextElem (Proto Point)) -> IO (Proto RouteSummary)

It might be more convenient to define a custom Handler monad stack in which we have access to the DB at all times:

newtype Handler a = WrapHandler {
      unwrapHandler :: ReaderT DB IO a
    }
  deriving newtype (Functor, Applicative, Monad, MonadIO, MonadReader DB)

runHandler :: DB -> Handler a -> IO a
runHandler db = flip runReaderT db . unwrapHandler

The types of our handlers then becomes

getFeature   :: Proto Point -> Handler (Proto Feature)
listFeatures :: Proto Rectangle -> (NextElem (Proto Feature) -> IO ()) -> Handler ()
recordRoute ::  IO (NextElem (Proto Point)) -> Handler (Proto RouteSummary)

Note that the callbacks to send or receive values still live in IO. The DB argument now disappears from methods also:

methods :: Methods Handler (ProtobufMethodsOf RouteGuide)
methods =
      Method (mkNonStreaming    getFeature  )
    $ Method (mkServerStreaming listFeatures)
    $ Method (mkClientStreaming recordRoute )
    $ Method (mkBiDiStreaming   routeChat   )
    $ NoMoreMethods

The only requirement from grapesy is that at the top-level we can hoist this monad stack into IO, using hoistMethods:

hoistMethods :: (forall a. m a -> n a) -> Methods m rpcs -> Methods n rpcs

Here’s how we can run the server:

runServerWithHandlers def config $ fromMethods $
  hoistMethods (runHandler db) methods

On the client

For the high-level API there is support for custom monad stacks also. One reason why you might want to do this is to avoid having to pass the Connection object around all the time. In the Basics tutorial our client functions had these signatures:

getFeature   :: Connection -> IO ()
listFeatures :: Connection -> IO ()
recordRoute  :: Connection -> IO ()
routeChat    :: Connection -> IO ()

Like on the server, we can define a custom monad stack to reduce syntactic overhead:

newtype Client a = WrapClient {
      unwrapClient :: ReaderT ClientEnv IO a
    }
  deriving newtype (Functor, Applicative, Monad, MonadIO, MonadCatch, MonadThrow, MonadMask)

data ClientEnv = ClientEnv {
      conn :: Connection
    }

In order for such a monad stack to be useable, it needs to implement MonadIO and MonadMask, as well as CanCallRPC; it’s this last class that tells grapesy to get get access to the Connection object:

instance CanCallRPC Client where
  getConnection = WrapClient $ conn <$> ask

With this defined, we can now avoid having to pass the connection around at all. Instead of importing from Network.GRPC.Client.StreamType.IO we import from Network.GRPC.Client.StreamType.CanCallRPC instead, which gives us a different definition of nonStreaming and friends. For example, here is getFeature:

getFeature :: Client ()
getFeature = do
    let req = ..
    resp <- nonStreaming (rpc @(Protobuf RouteGuide "getFeature")) req
    liftIO $ print resp

As for the server handlers, the callbacks provided to send and receive messages still live in IO; this means that we’ll need to liftIO them where appropriate:

listFeatures :: Client ()
listFeatures = do
    let req = ..
    serverStreaming (rpc @(Protobuf RouteGuide "listFeatures")) req $ \recv -> liftIO $
      NextElem.whileNext_ recv print

Using conduits

We discussed the simplest form of serverStreaming and co when we discussed the implementation of the client in the Basics tutorial, and we have also seen the generalized form to arbitrary monad stacks. There is one more form, provided in Network.GRPC.Client.StreamType.Conduit, which provides an API using conduits. You can find this example in tutorials/conduit; there is currently no conduit support on the server side.

The main idea is that serverStreaming provides a source to stream from, and clientStreaming_ provides a sink to stream to:

listFeatures :: Connection -> IO ()
listFeatures conn = do
    let req = ..

    let sink :: ConduitT (Proto Feature) Void IO ()
        sink = ..

    serverStreaming conn (rpc @(Protobuf RouteGuide "listFeatures")) req $ \source ->
      runConduit $ source .| sink

recordRoute :: Connection -> IO ()
recordRoute conn = do
    let source :: ConduitT () (Proto Point) IO ()
        source = ..

    resp <- clientStreaming_ conn (rpc @(Protobuf RouteGuide "recordRoute")) $ \sink ->
              runConduit $ source .| sink
    print resp

In bidirectional streaming finally we get two conduits, one in each direction (that is, one source and one sink).

(Ab)using Trailers-Only

For this final section we need to consider some more low-level details about how gRPC is sent over HTTP2. When we discussed final elements, we mentioned that gRPC messages are sent using HTTP2 DATA frames, but we didn’t talk about headers. In general, a gRPC request looks like this:

1. One or more HEADERS frames, containing the request headers. One of the most important headers here is the :path (pseudo) header, which indicates which RPC we want to invoke; for example, this might be /routeguide.RouteGuide/ListFeatures.

2. One or more DATA frames, the last of which is marked END_STREAM. We discussed these before.

This is probably as expected, but the structure of the response may look a bit more surprising:

1. Just like the request, we first get one or more HEADERS. An important example here is the content-type response header, which indicates what kind of message encoding is being used (for example, application/grpc+proto for Protobuf).

2. One or more DATA frames, the last of which is marked END_STREAM.

3. Finally, another set of headers, also known as trailers. This set of trailers provides some concluding information about how the RPC went; for example, if the RPC failed, then the trailers will include a grpc-status header with a non-zero value. Any application specific response trailing metadata (such as the checksum we discussed in the file server example) is included here as well.

There is however a special case, known as Trailers-Only: if there is no data to be sent at all, it is possible to send only HEADERS frames, the last of which is marked END_STREAM, and no DATA frames at all. Put another way, the two sets of headers (headers and trailers) are combined, and the data frames are omitted entirely.

The gRPC specification is very explicit about the use of Trailers-Only, and states that it can be used only in RPCs that result in an error:

Most responses are expected to have both headers and trailers but Trailers-Only is permitted for calls that produce an immediate error.

In grapesy this will happen automatically: if a server handler raises an error, and no outputs have as yet been sent to the client, then grapesy will automatically take advantage of Trailers-Only and only send a single set of headers.

However, some gRPC servers also make use of Trailers-Only in non-error cases, when there is no output (e.g. for server-side streaming). Since this does not conform to the gRPC specification, grapesy will not do this automatically, but it is possible if really needed. In tutorials/trailers-only you can find an example RouteGuide server which will take advantage of Trailers-Only in the listFeatures method, when there are no features to return:

listFeatures :: DB -> Call (Protobuf RouteGuide "listFeatures") -> IO ()
listFeatures db call = do
    r <- recvFinalInput call
    case featuresIn db r of
      [] -> sendTrailersOnly call NoMetadata
      ps -> StreamElem.forM_ ps NoMetadata (sendOutput call)

The difference between this implementation and the previous one can only be observed when we look at the raw network traffic; the difference is not visible at the gRPC level. Since this violates the specification, however, it’s possible (though perhaps unlikely) that some clients will be confused by this server.

Future work

The gRPC specification is only the core of the gRPC ecosystem. There are additional features that are defined on top, some of which are supported by grapesy (see list of features at the start of this post), but not all; the features that are not yet supported are listed below. Note that these are optional features, which have various degrees of support in the official clients and servers. If you or your company needs any of these features, we’d be happy to discuss options; please contact us at info@well-typed.com.

• Authentication. The gRPC Authentication Guide mentions three ways to authenticate: SSL/TLS, Application Layer Transport Security (ALTS) and token-based authentication, possibly through OAuth2. Of these three only SSL/TLS is currently supported by grapesy.

• Interceptors are essentially a form of middleware that are applied to every request, and can be used for things like metrics (see below).

• Custom Backend Metrics. There is support in grapesy for parsing or including an Open Request Cost Aggregation (ORCA) load report in the response trailers through the endpoint-load-metrics-bin trailer, but there is otherwise no support for ORCA or custom backend metrics in general.

• Load balancing. There is very limited support for load balancing in the ReconnectPolicy, but we have no support for load balancing as described in the Custom Load Balancing Policies Guide.

• Custom name resolution.

• Automatic deadline propagation. There is of course support for setting timeouts, but there is no support for automatic propagation from one server to another, adjusting for clock skew. See the section “Deadline Propagation” in Deadlines Guide for server.

• Introspection, services that allow to query the state of the server:

  • Admin services, used by tools such as grpcdebug.

  • Heath checking.

  • Reflection.

  • OpenTelemetry. We do support parsing or including a trace context in the grpc-trace-bin request header, but do not otherwise provide any support for OpenTelemetry yet.

• True binary metadata. There is support in grapesy for sending binary metadata (in -bin headers/trailers), using base64 encoding (as per the spec). True binary metadata is about avoiding this encoding overhead.

• Sending keep-alive pings (this will require adding this feature to the http2 library).

• Retry policies. The gRPC documentation currently identifies two such policies: request hedging, which sends the same request to a number of servers, waiting for the first response it receives; and automatic retries of failed requests. There is support in grapesy for the grpc-previous-rpc-attempts request header as well as the grpc-retry-pushback-ms response trailer, necessary to support these features.

Footnotes