Denys Miasnychenko

Software Architect / DevOps

Nav

epm: watching an Elasticsearch cluster from the terminal

epm (Elasticsearch Performance Monitor) is a terminal dashboard for an Elasticsearch cluster. It is a single Go binary: you point it at a cluster URL and it shows health, resource usage, indexing and search rates, latencies, and per-index and per-node tables, refreshed on an interval. It is MIT-licensed and builds for macOS and Linux on amd64 and arm64.

The problem

Most of my day-to-day questions about an Elasticsearch cluster are short: is it OK right now, and which index or node is hot? The usual tools don’t answer them well.

Kibana Stack Monitoring can, but it needs a browser, a Kibana, and usually a separate monitoring cluster. The _cat APIs are fast and available everywhere, but they only give you a snapshot. You can’t get a rate from them. And the counters in _stats and _nodes/stats are cumulative: index_total and query_total keep growing from the moment a shard or node started. “This index has done 4 billion index operations” tells you nothing about what it is doing now. I want operations per second and milliseconds per operation over the last few seconds.

I had already built this once as a Chrome extension. epm started as a port of that extension’s logic to Go, so I could get the same numbers in a terminal over SSH, on a jump host with no browser. The implementation plans in the repo say the metric calculations had to match the extension exactly.

What it shows

This is the layout sketch from the README:

┌──────────────────────────────────────────────────────────────────┐
│ my-cluster  ● GREEN  Connected  Last: 14:32:05  Poll: 10s        │
├──────────────────────────────────────────────────────────────────┤
│ GREEN │ 5 Nodes │ 42 Idx │ 210 Shards │ CPU 34% │ JVM 67% │ S 45%│
├──────────────────────────────────────────────────────────────────┤
│ Indexing Rate  │ Search Rate  │ Index Latency  │ Search Latency  │
│  1,204.3 /s    │   892.1 /s   │   2.34 ms      │   5.67 ms       │
│  ▁▂▃▅▇█▇▅▃▂    │  ▁▃▅▇▅▃▁▂▃   │  ▁▁▂▂▃▃▂▁▁▁    │  ▁▂▃▂▁▁▂▃▄▃     │
├──────────────────────────────────────────────────────────────────┤
│ Index Statistics  [/: search]  [1-9: sort col]  [←→: page] 1/5   │
├──────────────────────────────────────────────────────────────────┤
│ Node Statistics                                         Page 1/1 │
└──────────────────────────────────────────────────────────────────┘

From top to bottom:

Cards change color when a threshold is crossed: CPU above 80% (yellow) or 90% (red), JVM heap above 75% or 85%, storage above 80% or 90%, index latency above 500 ms, search latency above 1000 ms. There is no alert history. A red card only tells you to go and look.

Rates from cumulative counters

Every poll produces a snapshot. Rates and latencies are computed from the difference between two consecutive snapshots:

rate    = (ops_now - ops_prev) / seconds_between_polls
latency = (time_ms_now - time_ms_prev) / (ops_now - ops_prev)

That makes latency the average time per operation during the last interval. Dividing the raw counters would give you the lifetime average instead. On the first poll there is no previous snapshot, so the cards and rate columns show ---. Real values appear after the second poll.

Some details in this calculation are easy to get wrong:

Six endpoints, no client library

epm reads everything it needs from six endpoints:

The JSON endpoints use filter_path to request only the fields epm uses, and the _cat endpoints use h= to request only the needed columns. On a cluster with many indices, an unfiltered _stats response is large, and epm would throw most of it away. Response bodies are also capped at 32 MB, so a misbehaving endpoint cannot make the process read forever.

All of them are fetched in parallel with errgroup on every poll. The first five are required: if any of them fails, the poll fails. _cat/allocation, which supplies the node table’s Shards and Disk% columns, runs outside the error group and is non-fatal. If it fails or is not supported, those columns show --- and the rest of the dashboard keeps working.

There is no Elasticsearch client library. The HTTP client is net/http with a cloned default transport, and the response types are plain Go structs. The dependencies are Bubble Tea, Lip Gloss and Bubbles for the TUI, golang.org/x/sync for errgroup, go-runewidth for column widths, and testify for tests. The endpoints I need have been stable since 6.x, so a client library would add a lot of code and version-compatibility worries and give me nothing I need. The README lists 6.x, 7.x, 8.x and 9.x as tested.

The code lives in a few packages under internal/:

When the cluster goes away

Each fetch has a timeout of the poll interval minus 500 ms, capped at 10 seconds, so a slow cluster cannot stack up overlapping polls. When a poll fails, the header switches to DISCONNECTED with a short error class and a countdown to the next retry, and the last good data stays on screen. Retries back off exponentially: 2, 4, 8, 16, 32 seconds, then 60 seconds. If the error looks like a TLS certificate problem, the header suggests --insecure, and the same hint is printed again after the TUI exits.

Strings that come from the cluster, such as cluster, index and node names and error messages, are stripped of control characters and terminal escape sequences before they are rendered. Index names are user-controlled, and printing one raw to a terminal is a way to inject escape codes.

Analytics and write actions

Later releases added features that do more than watch the cluster.

The analytics screen (a) shows recommendations derived from the current snapshot, with no extra API calls. Each one is labelled [CRITICAL], [WARN] or [OK]. Among the checks:

The write actions came in v0.3.0, and they mean epm is not a read-only tool. It can delete indices.

To delete indices, Space selects rows in the index table, d opens a confirmation screen that lists every index to be deleted with WARNING: This action cannot be undone., and only y proceeds. The request is a single DELETE /<names> with each name path-escaped. Selection and deletion only work when the index table has focus and search mode is off.

For index settings, e opens a form for the selected indices (or the one under the cursor) with 11 dynamic settings: replicas, refresh interval, the six routing allocation include/exclude/require fields for node names and IPs, total shards per node, total fields limit, and read_only_allow_delete. The form loads current values from the cluster. When several indices are selected, the values are pre-filled from the first one and the form says so. Ctrl+S sends only the fields that changed, as one PUT /<names>/_settings. An unchanged form sends nothing.

There is no read-only mode switch. To keep epm strictly read-only, run it as a user whose role cannot delete indices or update settings.

Credentials

Credentials can be given in three ways, with a fixed priority: --user and --password flags, then ES_USER and ES_PASSWORD environment variables, then credentials embedded in the URI. The flags are there mostly because passwords with #, ? or % break URL parsing.

The URI credentials are removed from the stored base URL after parsing, so they are not echoed back in the header or in errors.

epm refuses to send credentials over plain http://. It exits with an error unless you pass --allow-insecure-auth, and prints a warning even when you do. For clusters with self-signed certificates there is --insecure, which skips TLS verification. I kept these as two separate flags because skipping certificate verification and sending a password in clear text are different risks.

Install and run

With Go:

go install github.com/jtsunne/epm-go/cmd/epm@latest

With Homebrew, from my tap (it is a formula, not a cask):

brew install jtsunne/tap/epm

Pre-built archives for macOS and Linux (amd64 and arm64) are on the releases page.

Then:

# local cluster, no auth
epm http://localhost:9200

# remote cluster, password with special characters, poll every 30 seconds
epm --interval 30s --user elastic --password 's3cr#t!' https://es.example.com:9200

# credentials from the environment
ES_USER=elastic ES_PASSWORD=changeme epm https://es.example.com:9200

The poll interval defaults to 10 seconds and must be between 5 and 300 seconds. Flags go before the URI; epm rejects a flag placed after it instead of silently ignoring it.

How it was built

epm was written in about a week at the end of February 2026, in phases, each with a written plan that is kept in docs/plans/completed/: foundation, engine, TUI shell, sparklines, tables, polish, UX improvements, then separate plans for credentials, the release pipeline, analytics, deletion, allocation columns and the settings editor. I built it with AI coding agents, and the git history shows the loop: implement a plan, run a code review, commit the fixes.

If I did a project like this again, I would keep the test requirement. Each plan task came with its tests, the later plans say outright that a task without tests is not done, and CI runs go vet and go test -race on every push to main and every pull request. The repo has about 5,700 lines of Go and 7,200 lines of tests. For code where the important part is arithmetic on counters, that ratio is why I trust the numbers.

The hardest bugs were in layout rather than logic. Lip Gloss’s Width() sets a minimum width, not a maximum, so a long cluster name made the header wrap onto a second line. The overview cards had different numbers of content lines, and without an explicit height the shorter cards were padded with lines that had no background color.

Releases are built by GoReleaser from GitHub Actions on a tag: static binaries with CGO_ENABLED=0, archives with checksums, and an updated formula pushed to the Homebrew tap.

Limits

epm watches one cluster per process. It keeps nothing on disk: the history is the last 60 polls in memory and disappears when you quit. Thresholds are fixed in the code and only change colors; there is no alerting. If you need history, alerting or many clusters, use a real monitoring stack. epm is for when you have a terminal open and want to know what the cluster is doing right now.