Real-Time Telemetry with Eventlog Live

Wen Kokke – Thursday, 24 September 2026

all open-source performance profiling telemetry  

Well-Typed are happy to announce Eventlog Live, a program that streams real-time telemetry from any Haskell application to any observability platform that supports the OpenTelemetry protocol, such as Grafana Cloud, Prometheus, or local viewers such as otel-tui and otel-desktop-viewer.

In this blog post, we’ll show a variety of ways to use Eventlog Live. For each showcase, we’ll link to the relevant instructions using Eventlog Live version 0.10.0.0 as well as the script that was used to generate the showcase. If you would like to use Eventlog Live, we recommend following the up-to-date instructions in the README of the latest release of Eventlog Live.

Heap Profiles by Closure Type

An animated GIF that shows a screen recording of a Grafana dashboard called Heap Profiles. It shows the heap profile of the example program called 'oddball', which has highly volatile memory usage. The dashboard has several configuration options, which let you select the service name and instance ID. For the example, the selected service name and instance ID are 'oddball' and a generated UUID, respectively. The dashboard is configured to show the last 5 minutes of data and refresh every 5 seconds. The dashboard is split into two columns. The left column contains a stacked area graph labelled 'Heap Profile' that shows a heap profile broken down by closure type. Below, there are three buttons labelled 'Start', 'Stop', and 'Census'. The right column contains six visualisations in a two-by-three grid. In order, these are labelled 'Heap Size', 'Memory Management', 'Heap Copied per GC by Generation', 'Heap Fragmentation by Generation', 'Heap Slop by Generation', and 'Heap Allocation by Capability'. The 'Heap Size' visualisation is a bar chart that shows three metrics. The 'Allocated from OS' metric shows the current size of the heap measured in megablocks, which is the unit in which the GHC RTS requests memory from the OS. The 'Allocated' metric shows the current size of the heap measured in blocks, which is the subunit that the GHC RTS uses to dole out memory to the application. The 'Live' metric shows the current size of the live heap, measured in live blocks. All three measurements are visualised in bytes. The 'Memory Management' visualisation shows three metrics that track the interaction with the OS: 'Current', 'Needed', and 'Returned'. The 'Current' metric shows the number of megablocks currently held by the RTS. The 'Needed' metric shows the number of megablocks that the RTS currently needs. If 'Needed' exceeds 'Current', the RTS is about to request more memory from the OS. If 'Needed' is less than 'Current', the RTS should return the excess megablocks to the OS. If this happens, the number of returned megablocks is shown in the 'Returned' metric. The 'Heap Copied per GC by Generation', 'Heap Fragmentation by Generation', and 'Heap Slop by Generation' are bar charts that show the number of bytes copied during GC, the amount of heap fragmentation found during GC, and the amount of slop bytes found during GC, respectively. These metrics are broken down by heap generations. In the example, these are Generation 0 and Generation 1. The 'Heap Allocation by Capability' visualisation is a bar chart that shows the number of bytes allocated by each capability. This metric is broken down by capability, but in the example there is only one Capability.

The Grafana Heap Profiles dashboard, which shows a heap profile broken down by closure type (-hT) as well as a variety of metrics for memory usage. For detailed instructions, see Eventlog Live – The Basic Version. This showcase uses the oddball-with-pipe example with the Docker Compose setup in demo/docker-compose.yml.

Eventlog Live

Eventlog Live analyses your application’s eventlog and exports it via the OpenTelemetry protocol. Eventlog Live is lightweight, runs side-by-side with your application in only a few megabytes of memory, and requires no instrumentation other than the -threaded and -rtsopts build options.1 All you need is somewhere to send your data. For instance, if you have a Grafana Cloud2 account, adding telemetry to your application is as easy as:

# Create a pipe for the eventlog
EVENTLOG_PIPE="/tmp/eventlog.pipe"
mkfifo "${EVENTLOG_PIPE}"

# Start your application
your-application +RTS -l -ol"${EVENTLOG_PIPE}" -hT --eventlog-flush-interval=1 -RTS &

# Start eventlog-live-otlp
OTEL_SERVICE_NAME="your-application"                                              \
OTEL_RESOURCE_ATTRIBUTES="service.instance.id=$(uuidgen)"                         \
OTEL_EXPORTER_OTLP_ENDPOINT="https://otlp-gateway-<YOUR_REGION>.grafana.net/otlp" \
OTEL_EXPORTER_OTLP_HEADERS="Authorization=Basic%20<YOUR_AUTHORIZATION_HEADER>"    \
eventlog-live-otlp --eventlog-file="${EVENTLOG_PIPE}" -hT --eventlog-flush-interval=1

Out of the box, Eventlog Live supports -hT and -hi heap profiles3 as well as a variety of memory usage metrics, cost-centre stack profiles, log messages, productivity, detailed thread & capability usage traces. Each of these features is showcased below using an example Grafana dashboard. These dashboards can be found in the repository under demo-grafana-cloud/grafana-dashboards and demo/config/grafana-dashboards. Click any showcase for more information.

Heap Profiles by Info Table

An animated GIF that shows a screen recording of the same Heap Profiles dashboard as under Heap Profiles by Closure Type, except that the categories for each heap profile sample show more detailed information based on the info tables.

The Grafana Heap Profiles dashboard, which shows a variety of metrics for memory usage and a heap profile broken down by info table (-hi). For detailed instructions, see Eventlog Live with Heap Profiling by Info Table. This showcase uses the oddball-with-hi example with the Docker Compose setup in demo/docker-compose.yml.

Cost-Centre Stack Profiles

An animated GIF that shows a screen recording of a Grafana dashboard called Cost-Centre Stack Profiles. The dashboard has several configuration options, which let you select the service name and instance ID. For the example, the selected service name and instance ID are 'jumpy-jump' and a generated UUID, respectively. The dashboard is configured to show the last 6 hours of data and refresh every 5 seconds. The dashboard has one visualisation, which is split into two columns. The left column shows a table of symbols together with their sample count. The top two samples are IDLE and SYSTEM, followed by uniformIntegralM, an ever-changing list of ten functions called jumpyJump0 to jumpyJump9, and, occasionally, PROFILING:OVERHEAD_OF. The right column shows the stack profile as a flame graph. (Technically, an icicle chart, since the spikes go downward, but I don't think people use that word much anymore.) The flame graph is dominated by the IDLE category, which is sampled, roughly, two orders of magnitude more often than every other category.

The Grafana Cost-Centre Stack Profiles dashboard, which shows the cost-centre stack samples collected by GHC's built-in profiler in a flame graph. For detailed instructions, see Eventlog Live with Cost-Centre Stack Profiling. This showcase uses the jumpy-jump-with-cost-centre-profiler example with the Docker Compose setup in demo/docker-compose.yml.

Logs

An animated GIF that shows a screen recording of a Grafana dashboard called Logs. The dashboard has several configuration options, which let you select the service name and instance ID. For the example, the selected service name and instance ID are 'oddball' and a generated UUID, respectively. The dashboard is configured to show the last 6 hours of data and refresh every 5 seconds. The dashboard has one visualisation, which shows a list of log messages with their time and severity. Each log message has the TRACE severity and is some variant of 'Summing 313172 numbers', except the first two messages which are the thread labels for the timer and IOManager threads emitted by the RTS.

The Grafana Logs dashboard, which shows log messages emitted using traceEventIO, traceMarkerIO, and threadLabel. This showcase uses the oddball-with-pipe example with the Docker Compose setup in demo/docker-compose.yml.

Logs as Markers

An animated GIF that shows a screen recording of the same Heap Profiles dashboard as under Heap Profiles by Info Table. However, in the configuration options a switch labelled UserMarker is enabled and the field labelled Filter UserMarker contains the regular expression 'Summing \d+ numbers', and each visualisation shows dotted red lines at the times that correspond to these markers.

The Grafana Heap Profiles dashboard, which shows dotted red lines on each visualisation at the times that correspond to the markers, emitted using traceMarkerIO, that match the regular expression Summing \d+ numbers. This showcase uses the oddball-with-hi example with the Docker Compose setup in demo/docker-compose.yml.

Productivity

An animated GIF that shows a screen recording of a Grafana dashboard called Threads. The dashboard has several configuration options, which let you select the service name and instance ID. For the example, the selected service name and instance ID are 'oddball' and a generated UUID, respectively. The dashboard is configured to show the last 5 minutes of data and refresh every 5 seconds. The dashboard is split into two columns. In the left column, there is a bar chart labelled 'Productivity by Capability', which shows the ratio between executing user code and garbage collection as a percentage, split out by capability and summed over the entire execution. Oddball uses only one capability, Capability 0. From the bar chart, we can see that this capability is about 80% productive during the first few seconds, then nosedives to about 65%, and stays there for the remainder of the animation. In the right column, there is a stacked area graph labelled 'Capability Usage', which shows the capability usage over a sliding window, split out into executing user code (labelled 'Mutator'), garbage collection (labelled 'GC'), and idle time (labelled 'Idle'). While this affirms the image shown by the previous visualisation, it reveals that oddball is idle the vast majority of the time.

The Grafana Threads dashboard, which shows the cumulative productivity per capability, as well as the overall productivity over a sliding window. This showcase uses the oddball-with-pipe example with the Docker Compose setup in demo/docker-compose.yml.

Thread & Capability Usage

A screenshot that shows the Grafana Trace Explorer, which is currently exploring a thread usage trace for thread 3. Each span is listed with its type (either Running or Blocked) and its duration. One particular span is selected, which unfolds the entry and reveals more information. Notably, it reveals that thread 3 was blocked during this span due to a foreign function call.

Eventlog Live’s OTLP exporter is configured using OpenTelemetry’s environment variables. The output of eventlog-live-otlp --help has comprehensive documentation on its support for these environment variables. By default, Eventlog Live aggregates telemetry data over 1 second intervals and exports every 30 seconds, but these intervals can be configured for each telemetry signal using configuration files.

Dynamic Control with Eventlog Socket

If you instrument your application with eventlog-socket, you’ll be able to start/stop profiling while your application is running using a REST API, which you can connect, e.g., to buttons on your Grafana dashboard, as we’ve done on the example Heap Profiles dashboard.

Dynamic Control with Eventlog Socket

An animated GIF that shows a screen recording of the same Grafana Heap Profiles dashboard that was described above. It continues to show the heap profile of the 'oddball' example program. This recording illustrates the usage of the 'Start' and 'Stop' buttons. At the start of the recording, the 'Heap Profile' visualisation shows a history of volatile and spiky memory usage. After a few refreshes, the 'Stop' button is pressed, the message 'Requested Heap Profiling Stop' appears in the top-right corner, and the 'Heap Profile' visualisation flatlines. A few refreshes later, the 'Start' button is pressed, the message 'Requested Heap Profiling Start' appears in the top-right corner, and the 'Heap Profile' visualisation resumes. The 'Census' button can be used to request a single heap census, though this isn't shown in the recording. The visualisations in the right column are unaffected by the 'Start' and 'Stop' button, and continue to update during the entire recording.

Using the buttons on the Grafana Heap Profiles dashboard. When “Stop” is pressed, the heap profile flatlines. When “Start” is pressed, the heap profile resumes. The “Census” button can be used to request a single heap census. For detailed instructions, see Eventlog Live with Eventlog Socket. This showcase uses the oddball-with-hT example with the Docker Compose setup in demo/docker-compose.yml.

Eventlog Socket lets your application write its eventlog over sockets and uses the other direction of communication for control messages. Eventlog Socket’s control protocol is generic. It isn’t bound to any specific application or library. Any library can register its own control commands under its own namespace,4 using Haskell IO actions as callbacks:

registerMyEventlogSocketSupport = do
  myPackageNamespace <- registerNamespace "my-package-name"
  registerCommand myPackageNamespace (CommandId 1) myCommandCallback1
  registerCommand myPackageNamespace (CommandId 2) myCommandCallback2

Any command that your application registers with Eventlog Socket can be invoked using Eventlog Live’s REST API.

For security reasons, Eventlog Socket’s control protocol and Eventlog Live’s REST API are hidden behind the +control feature flag, which is disabled by default. If you develop a library that (optionally) uses Eventlog Socket’s control protocol, we recommend using the same +control feature flag.

Call-Stack Profiles with GHC Stack Profiler

If you instrument your application with ghc-stack-profiler, you’ll be able to use its lightweight call-stack profiler.

Call-Stack Profiles with GHC Stack Profiler

An animated GIF that shows a screen recording of a Grafana dashboard called Call-Stack Profiles. This dashboard has the same layout as described under Cost-Centre Stack Profiler, but the profiles are slightly different. Notably, the IDLE, SYSTEM, and PROFILING:OVERHEAD_OF categories are not recorded by GHC Stack Profiler. The most prevalent category is 'other' followed by the same ever-changing list of ten functions called jumpyJump0 to jumpyJump9. The flame graph is significantly spikier. The IDLE category isn't recorded and does not dominate the flame graph, which leaves a lot more room for the deeply nested calls jumping between the various jumpyJump functions.

The Grafana Call-Stack Profiles dashboard, which shows the call-stack samples collected by ghc-stack-profiler in a flame graph. For detailed instructions, see Eventlog Live with GHC Stack Profiler. This showcase uses the jumpy-jump-with-ghc-stack-profiler example with the Docker Compose setup in demo/docker-compose.yml.

GHC Stack Profiler samples GHC’s runtime call-stack, rather than a virtual cost-centre stack, and writes these samples to the eventlog. Based on our benchmarks, GHC Stack Profiler has virtually no overhead when it’s not running and we observed 2-8% overhead while it was running, compared to GHC’s built-in cost-centre stack profiler, where we observed 54-128% overhead while it wasn’t running and 98-136% overhead while it was running.

If you’ve also instrumented your application with Eventlog Socket, you’ll be able to start/stop GHC Stack Profiler while your application is running using the same REST API, which you can connect, e.g., to buttons on your Grafana dashboard, as we’ve done on the example Call-Stack Profiles dashboard.

GHC Stack Profiler can also be used without Eventlog Live, e.g., by exporting the profiles to speedscope. We’ll discuss GHC Stack Profiler in detail in an upcoming blog post.

Local Viewers

Eventlog Live does not yet include a built-in telemetry viewer, as it was primarily developed to monitor long-running processes in production. We are currently developing a built-in telemetry viewer, but the prototype is not yet ready for publication. In the meantime, some developers may feel that using either Grafana Cloud or the Docker Compose setup in demo/docker-compose.yml is too heavyweight for local development. Fortunately, the OpenTelemetry ecosystem has several lightweight telemetry viewers, such as otel-tui and otel-desktop-viewer, which are general purpose telemetry viewers that use a text and browser-based UI, respectively.

Eventlog Live with otel-tui

An animated GIF that shows a screen recording of a terminal session. The terminal is split into three screens. On the left, taking up the majority of the terminal, is a screen labelled 'otel-tui'. On the right, split vertically, are two screens labelled 'oddball' and 'eventlog-live-otlp'. The 'otel-tui' screen is itself split into three subscreens. A narrow column on the left, labelled 'Metrics', lists all incoming metrics by name. These names correspond to the names described under Heap Profile by Closure Type, except formatted in camel-case and prefixed with 'ghc_eventlog_'. Currently, the metric named 'ghc_eventlog_HeapLive' is selected. The wide column on the right is split vertically. The top row, labelled 'Details', shows the exact structure of the selected OTLP message for the selected metric. The bottom row, labelled 'Chart', shows a text rendering of a line chart that shows the values of the 'ghc_eventlog_HeapLive' metric. This shows a spiky line, which jumps between 10-30MB, though the Y-axis is labelled in bytes rather than megabytes. The 'oddball' screen shows the output of the oddball program, which consists of many lines saying 'Generating X random numbers' and 'Sum: Y' for ever-changing values of X and Y. The 'eventlog-live-otlp' screen shows the debug output of Eventlog Live, which consists of many lines saying '[DEBUG] Received X events.', '[DEBUG] Exported Y metrics.', and '[DEBUG] Exported Z logs.', for ever-changing values of X, Y, and Z.

This showcase uses the oddball-with-otel-tui example with otel-tui version 0.7.4. This example creates a screen session that combines otel-tui with oddball and eventlog-live-otlp.

Eventlog Live with otel-desktop-viewer

An animated GIF that shows a screen recording of the browser-based otel-desktop-viewer application. The website is split into three columns. The left column has three tabs labelled 'Traces', 'Metrics', and 'Logs'. Currently, the 'Metrics' tab is selected, which lists all incoming metrics by name and application. These names correspond to the names described under Heap Profile by Closure Type, except formatted in camel-case and prefixed with 'ghc_eventlog_'. Currently, the metric named 'ghc_eventlog_HeapProfSample (oddball)' is selected. The right column lists all series in this metric. The 'ghc_eventlog_HeapProfSample' metric is the heap profile, broken down by closure type, so the various series correspond to the various closure types. Only the top few series are selected. Amongst these are built-in closure types such as FUN, FUN_2_0, STACK, and BLACKHOLE, as well as user-defined function types such as ghc-bignum:GHC.Num.Integer.IS. The series are constantly reordered, so they're a bit hard to read. The middle column, which takes up about half of the website, shows a chart with the selected series. While ten series are selected, only two lines are visible. STACK and ghc-bignum:GHC.Num.Integer.IS. This is because these samples represent the vast majority of the heap size used by the oddball program. The remaining lines, presumably, all hug the X-axis, though this isn't visually clear.

This showcase uses the oddball-with-otel-desktop-viewer example with otel-desktop-viewer version 0.5.0.

Future Work

We are currently developing a built-in telemetry viewer that is purpose-built for GHC’s telemetry, with the intention of making Eventlog Live more useful for short development sessions. We’re also planning to create a library for writing structured log messages to the eventlog, expand the analyses supported by Eventlog Live, and expand the Eventlog Socket protocol to permit command arguments. Moreover, we’re always open to suggestions and feature requests from the community.

Acknowledgements

This work has been performed in collaboration with Mercury, who have a long-term commitment to the scalability and robustness of the Haskell ecosystem. Well-Typed are always interested in projects and looking for funding to improve Haskell tools. Please contact info@well-typed.com if we might be able to work with you!

  1. If your application is built with GHC 9.2 or older, it must also be built with -eventlog. Since GHC 9.4, the eventlog is enabled by default.↩︎

  2. The repository contains a self-contained demo for using Eventlog Live with Grafana Cloud. Unfortunately, as Grafana Cloud changes its user interface frequently, that demo may not be fully up-to-date.↩︎

  3. Eventlog Live is developed and tested with the -hT and -hi heap profile breakdowns, but the -hm/-hd/-hy/-he heap profile breakdowns should work. The -hc/-hb/-hr heap profile breakdowns require additional support, which is planned but not yet implemented.↩︎

  4. We strongly recommend using your package name.↩︎