Metrics vs Logs: where it lives and what it costs

1 July 2026

TL;DR

• Two questions decide where a field belongs: do you want an aggregate or a specific instance, and is the field bounded or unbounded.
• A time-series database pays for series count, not datapoints. One unbounded label (a full request line) on a Prometheus metric would need roughly 85 GB of RAM for a single day of traffic.
• Elasticsearch forgives unbounded fields, but you pay for it in storage: on the same traffic, the metric costs MBs a day and the logs cost GBs, roughly 180x to 1,300x apart.
• An aggregate you have not materialized as a metric is recomputed on every read. How much that costs depends on the aggregate: a count in Elasticsearch took 4 ms, an avg took 758 ms.
• The default answer is the hybrid: a bounded metric in the TSDB for how much, the full detail in the log store for which ones, and shared bounded labels to jump between them.

This is how I decide where a piece of data belongs: a time-series database (Prometheus or InfluxDB) or a log/document store (Elasticsearch). I skip the usual “three pillars of observability” tour and stick to one decision axis and one cost reality, anchored in real HAProxy edge traffic from a mid-size e-commerce setup I worked with: about 25M requests a day across a couple of hundred sites.

A note on “TSDB” in this article: where the cost principle is shared I say TSDB, meaning both Prometheus and InfluxDB. Both pay for series count, not datapoint count, so the cardinality trap is identical in shape. Where a number is measured on one specific engine, it is labelled as such (the hard per-series RAM figure below is from Prometheus; InfluxDB’s index layout differs in detail but not in the direction of the cost).

Who this is for

If you currently store numeric values inside log lines and look at everything in Kibana, this article is for you. That setup works for a while. Then the storage bill climbs, the “just add a dashboard” request turns into a cluster-sizing conversation, and nobody can say why a graph that is conceptually a single number costs what it costs. The usual reason is that a metric and an aggregation got treated as the same thing, and a field that should have been a cheap counter got stored as millions of searchable documents instead.

Nothing here says Kibana is wrong or that you should rip anything out. What follows is a decision framework: given one field, one question you want to ask of it, decide where it belongs and what that choice costs. Two inputs drive every decision, and they fit in a 2x2.

The decision: one 2x2

Ask two questions about the field you are about to store.

Question A. Do you want an aggregate or a specific instance? An aggregate is a number over time: a count, a rate, a p95, “how many 5xx in the last 5 minutes.” A specific instance is one identifiable event: this one request, this one error, for this one customer.

Question B. Is the identifying field bounded or unbounded? Bounded means a finite set that does not grow with traffic: HTTP status code, region, a normalised route. Unbounded means the set grows with usage: user_id, request_id, client IP, full URL, trace_id.

Cross them:

Three of the four cells are easy. This article is mostly about the top-right cell, the trap, because that is the one that puts numeric data in the wrong store without anyone noticing and hands you either a surprise Elasticsearch bill or a dead Prometheus.

Cardinality, explained like you’re five

Every wrong call in this space traces back to misunderstanding one concept: cardinality.

The usual rule of thumb, “many unique values is bad,” is imprecise enough to mislead you. A more precise picture:

A metric is a labelled jar of counters. Cardinality is how many jars you have to keep on the shelf.

When a request comes in, Prometheus looks at the labels you attached to the metric, finds the one jar whose label matches, and clicks its counter up by one. Two costs live in that sentence, and they are wildly different:

• Clicking the counter is almost free. A datapoint is 1-2 bytes (delta-of-delta timestamp encoding + Gorilla XOR on the value). A million clicks into the same jar is a rounding error.
• Putting a new jar on the shelf is the expensive part. Each distinct jar is a new time series: a chunk held in memory, an entry in the inverted index, and ongoing ingestion bookkeeping. And the shelf is effectively permanent for the retention window.

So the only question that matters when you reach for a label is:

If I put this field on the shelf as a label, how many jars do I end up with, and does that shelf ever stop growing?

A field whose shelf stops growing is bounded. A field whose shelf grows with traffic is unbounded. Bounded fields make good metric labels. Unbounded fields are cardinality bombs, and they belong in a log or trace store, not on a metric.

The sections below check this against real edge traffic, so the numbers come from measurement instead of hand-waving.

The materialization check: who pays for the aggregate, and when

The 2x2 answers one question: where does a field belong. It does not answer a second, follow-up question that only applies once you have already landed in the “aggregate + bounded, this is a metric” cell: when does the counting happen. Getting the 2x2 right does not save you if you get this one wrong. Treat it as a check you run after the 2x2 rather than a second decision axis.

The check. Once a field is classified as aggregate + bounded, has the aggregate actually been materialized as a metric (one increment per event, written once at ingest), or is it still being recomputed query-time from raw documents on every request (an Elasticsearch aggregation standing in for a metric that was never built)?

This is the same jar analogy at a different moment in the story. Prometheus is a waiter who drops a coin in the right jar the instant a customer pays. By the end of the shift the jar already holds the total; reading it is just looking at a number. An Elasticsearch aggregation is the opposite: nobody counts tips as they come in, every receipt goes in a shoebox, and the total is recomputed by dumping out the shoebox and adding it up again. Ask for the total once, you add once. Ask ten times during the shift, you add up the whole shoebox ten times.

This matters even inside the cells I already called “easy.” Aggregate

• bounded says a field belongs on a metric. It does not say where the sum gets computed. If http_status_code (bounded, 25 values) only ever lives in the log store, and a dashboard panel computes avg(response_time) by status through an Elasticsearch aggregation on every refresh, that panel is recomputing the average from raw documents every single time someone opens it. The field was classified correctly. The compute is still in the wrong place, and you pay for it on every query instead of once per event.

Before wiring a panel to an aggregation query, ask: “Does this number get computed once, on write, or recomputed on every read?” Recomputed on every read means query-time cost. It does not disappear just because the field is bounded, but, as the measurements below show, it does not have one fixed size either. It depends on what kind of aggregate you asked for.

Measured: two aggregates, two verdicts

It would be convenient if the rule were “query-time in Elasticsearch is always slower than write-time in Prometheus.” It is not, and overstating it would be exactly the kind of hand-waved number this article exists to avoid. What drives the cost is what you are aggregating, not which engine you asked. Same HAProxy edge traffic (the same two load balancers, lb-1 and lb-2, that wrote the edge-logs-2026.06.29 index), same day, two different aggregates, three runs each, request cache disabled on the Elasticsearch side so nothing is hidden behind a cached answer.

Case 1: avg(response_time) by status, the worst case for query-time. This aggregate has to touch a numeric field on every matching document; there is no way to shortcut it from a doc count.

Measured in Elasticsearch, three consecutive runs, ?request_cache=false, the full day, about 25M documents:

POST edge-logs-2026.06.29/_search?request_cache=false
{ "size": 0, "aggs": { "by_status": {
    "terms": { "field": "http_status_code", "size": 50 },
    "aggs": { "avg_rt": { "avg": { "field": "response_time" } } } } } }

Server-side took: 807 ms, 773 ms, 694 ms (mean 758 ms). Every run scans doc values for all matching documents to compute the average.

This is also the case where the Prometheus setup there had no write-time equivalent to offer, and not by accident. The HAProxy exporter in use does not expose response time broken down by status code, and there is a reason a sane exporter would not: doing so means one time series per status-per-backend-per-instance combination, the exact cardinality multiplication this article warns against. A deliberately designed histogram or summary with bounded labels could exist; that is a schema and cardinality-budget decision someone would have to make on purpose, not something Prometheus is architecturally incapable of. Nobody had made that decision, because the natural way to expose this metric is exactly the cardinality bomb. avg(response_time) by status is expensive query-time in Elasticsearch and would cost a deliberate cardinality trade-off to make write-time. Neither path is free; Elasticsearch at least makes the question answerable today, without a schema change.

Case 2: count by status, the best case for Elasticsearch, and it still does not flip the outcome the way Case 1 might suggest. Counting documents in bounded ranges never touches the response_time field at all.

Measured in Elasticsearch, three consecutive runs, ?request_cache=false, same full day:

POST edge-logs-2026.06.29/_search?request_cache=false
{ "size": 0, "aggs": { "by_status_class": { "range": {
    "field": "http_status_code",
    "ranges": [ {"key":"1xx","from":100,"to":200}, {"key":"2xx","from":200,"to":300},
                {"key":"3xx","from":300,"to":400}, {"key":"4xx","from":400,"to":500},
                {"key":"5xx","from":500,"to":600} ] } } } }

Server-side took: 7 ms, 3 ms, 2 ms (mean 4 ms).

Measured in Prometheus, three consecutive runs, ?stats=all, same two load balancers, same day, 24h window ending at the day’s close:

GET /api/v1/query?query=sum by (code) (increase(
  haproxy_frontend_http_responses_total{
    instance=~"lb-1.*|lb-2.*"}[24h]))

Server-side execTotalTime: 54.7 ms, 46.8 ms, 45.1 ms (mean 48.9 ms).

Here Elasticsearch’s 4 ms beats Prometheus’s 48.9 ms, roughly 12x, on the exact same traffic. The shapes are close but not identical: five Elasticsearch ranges against six Prometheus series (1xx-5xx plus an other bucket the exporter also tracks and that stayed at zero throughout this measurement), close enough to call the same question without claiming a byte-for-byte match. increase() over a 24-hour window has to walk every sample in that window for those series; counting documents into five ranges is close to free for Elasticsearch’s index structures. The engine that “always wins” flips depending on the aggregate.

Read together, the two cases say this: write-time is fast because the cost was already paid, once, at ingest, regardless of which aggregate you will ask for later. It has nothing to do with Prometheus being a faster engine. Query-time cost in Elasticsearch varies with the aggregate: avg/percentile is expensive (it touches every matching document’s numeric field), while count/terms is cheap (it only touches the field you are grouping by). Neither “~758 ms” nor “~4 ms” is a property of Elasticsearch. The property that holds in either case is that an aggregate you have not materialized costs whatever it costs to compute on demand, every single time.

The multiplier developers hit: history window x panel count

Both cases above ran against a single day. The query a developer runs is “show me this graph for the last one or two weeks,” and a real dashboard has a screen full of graphs. Both variables make the cost worse, and they make it worse differently on each side.

First, the same two aggregates, widened to a real multi-week window. 12 of the available daily indices covering June 16 to 29, 2026 (June 20 and 21 had aged out of the cluster’s retention by the time of measurement, so the window is 12 real days, not a clean 14, stated exactly rather than rounded), about 320M documents total on the Elasticsearch side.

• Elasticsearch, same avg(response_time) by status query as Case 1, run across all 12 daily indices, ?request_cache=false, three consecutive runs: server-side took 9,353 ms, 8,976 ms, 9,082 ms (mean 9,137 ms, roughly 12x the one-day 758 ms, in line with roughly 12x the documents scanned).
• Elasticsearch, same count by status range aggregation as Case 2, same 12 daily indices, three consecutive runs: server-side took 47 ms, 17 ms, 9 ms (mean 24 ms, about 6x the one-day 4 ms, better than linear; counting into bounded ranges barely notices the extra volume).
• Prometheus, same count by status query as Case 2, widened to increase(...[288h]), 288 hours, matching the same 12 real days the Elasticsearch side scanned, evaluated at the same day’s close, three consecutive runs: server-side execTotalTime 580 ms, 575 ms, 576 ms (mean 577 ms, about 11.8x the one-day 48.9 ms, in line with the 12x larger window; increase() walks every sample in it, so a wider window is not free the way a bounded-range count is).

The count-by-status comparison does not flip back in Prometheus’s favor at this window either. On the same 12 days measured on both sides, Elasticsearch is still faster here, 24 ms vs 577 ms, ~24x, because range aggregations over an indexed numeric field barely care how many days are behind the query, while increase() has to walk every sample in a widening window. The avg-by-status comparison is unaffected by window size in a different way: there is still no Prometheus number to put next to the 9.1 seconds, because nobody has built that metric, and building it would mean making the cardinality trade-off from the start of this article.

Second, multiply by how many panels are on the screen. Take a realistic dashboard of 20 panels, a mix of the two aggregate shapes above: 5 panels doing avg/percentile-style math (response time, latency percentiles, byte-size averages, broken down by a bounded field) and 15 panels doing count/rate-style math (request volume, error rate, status breakdowns), all over the same 12-day window. This is backend query cost, not wall-clock render time. A real dashboard fires these queries concurrently and a fast frontend can overlap some of the wait, but every query still lands on a backend that has to do the work, and that total query load is what you are provisioning capacity for.

Derived from the per-panel numbers measured above, no additional queries run: in Elasticsearch, 5 x 9,137 ms + 15 x 24 ms ≈ 46 seconds of summed backend query time to answer all 20 panels. In Prometheus, the 15 count/rate panels cost 15 x 577 ms ≈ 8.7 seconds of summed query time, and the other 5 panels have no cost figure to add at all, because nobody has paid the cardinality price to build that metric. Prometheus could answer them in principle.

Multiplied out, the mistake shows up across a whole dashboard, not in one slow panel. A handful of avg/percentile panels dominate the total query load, and “just add a similar panel” adds tens of seconds of backend work rather than milliseconds, without anyone noticing. The fix is to decide per panel instead of moving everything to Elasticsearch or everything to Prometheus. A panel you will reload often, over a window that keeps growing, on a field you can keep bounded, belongs write-time: build the metric (a histogram or a summary, with labels kept to the bounded dimensions you will alert or dashboard on, route, method, a status class rather than the exact code if that keeps the label set small) and pay the cardinality cost once, deliberately, instead of paying a growing query-time cost every reload. A panel that is genuinely ad hoc, or whose field cannot be bounded without losing the point of asking (the exact request, the exact user, the long tail of a full request path), stays query-time in the log store, where the cost is real but at least does not depend on having predicted the question in advance.

A caveat, so I do not overstate the cache side either: Elasticsearch’s shard request cache can serve a repeated aggregation without recomputing it, and Elastic specifically recommends it for logging use cases, where it is a genuinely good fit. That fit depends on which half of your data you are asking about. Per the Elastic 8.19 docs, the cache is invalidated on every shard refresh, not on a timer, “making [the expire setting] less critical.” On the Elastic Stack setup measured here, index.refresh_interval defaults to 1 second for an index that has been searched in roughly the last 30 seconds (Elastic Cloud Serverless defaults differently, and a shard with no recent search traffic can skip background refreshes entirely, so this is a default under that config and that access pattern, not a law).

The two kinds of shard behave differently. A rolled-over, no-longer-written daily index, queried with the same absolute time range twice, stops refreshing once writes stop, so the cache holds and a repeated report can be served for free. edge-logs-2026.06.29 while it was still the active write index, queried through a rolling now-1h window by a live dashboard, gets neither condition: continuous HAProxy ingest keeps triggering refreshes on that actively-searched shard, and the window itself shifts on every dashboard reload, so even a byte-identical repeat would still land on the wrong side of the next refresh. The cache is a real feature once an index’s write path has gone quiet. It is not in play for a live dashboard watching today’s traffic, which is the case this section is about.

The verdict

Getting the 2x2 right tells you a field is a metric. The materialization check tells you whether it has been built as one, and the two cases above show that “not built as one” does not have a single price tag: the same Elasticsearch aggregation ran in 4 ms for a count and 758 ms for an avg, nearly 190x apart, on the same index, the same day. The one thing that does not vary is which side pays once: a write-time counter, whatever shape it takes, is paid for at ingest and never recomputed. Deciding whether that upfront cost is worth paying, including the cardinality cost of the labels a real metric would need, is the actual judgment call. This is also the reason to keep both stores instead of picking a winner: put the aggregates you will ask for repeatedly on the cheap, predictable write-time side, and leave Elasticsearch for the ad hoc and the genuinely unbounded, where its query-time cost is the price of not having to guess in advance what you will need to ask.

The same data, measured both ways

All numbers below come from one index, edge-logs-2026.06.29, a single day of HAProxy edge traffic for a mid-size e-commerce business running a couple of hundred storefront domains.

Measured in Elasticsearch (_cat/indices), full closed day: about 25M requests, about 12 GB primary store (about 24 GB with the one replica), ~500 bytes per document compressed (best_compression).

Bounded fields: small, fixed shelves (good metric labels)

These are the fields where the jar count is tiny and does not grow with traffic. Querying the distinct count with an Elasticsearch cardinality aggregation and the value set with a terms aggregation:

Query shape:

POST edge-logs-2026.06.29/_search
{ "size": 0, "aggs": {
    "distinct": { "cardinality": { "field": "http_status_code" } },
    "values":   { "terms": { "field": "http_status_code", "size": 50 } } } }

http_status_code is the textbook metric. The top of the distribution: 200 is about 83% of all requests, 307 about 9.5%, 301 about 1.7%, 204 about 1.4%, 404 about 1.1%, 403 about 0.9%, and down at the tail where you actually care, the 5xx handful. That is exactly the shape you want: rate(http_requests_total{status="5xx"}) is one cheap series you can alert on, and the shelf is 25 jars no matter how much traffic comes in.

Unbounded fields: shelves that never stop growing (cardinality bombs)

Same index, same day, same cardinality aggregation pointed at the high-variance fields:

Note: the cardinality aggregation is approximate (HyperLogLog++), accurate to within ~1% at this scale. That is good enough here, because the exact count does not matter when the answer is “millions.”

Why http_request explodes is visible in the raw values: path parameters, query strings, and cache-busting asset hashes are all unique per request:

/brand-tactical-flashlight-618fa-orange.html
/l/assets-7b3e90c25f1/base/catalog/desktop/img/rating-star-off.gif
/9e2c7a41d03b5f86e1a4/t.png

The fix is to normalise /api/order/12345 to a bounded route /api/order/:id. The route is a label; the full path is not.

Note on x_trace_id: this frontend captures the header but does not populate it (empty on 100% of docs in this index), so these edge logs have no per-request id to use as the example. http_request is the stronger one anyway; it is the cardinality bomb people actually create by accident.

What the bomb costs

Here the “datapoints are cheap, series are expensive” claim gets a measurement. Measured against an internal Prometheus (an in-cluster kube-prometheus-stack, reading its own self-metrics):

• prometheus_tsdb_head_series: about 0.9M active series
• process_resident_memory_bytes: about 7 GB resident
• so ~7.8 KB of RAM per active series (RSS / head series)
• on-disk chunk size averages ~126 bytes (datapoints really are cheap)

Now put the bomb on that scale. The single label http_request would mint about 11M series in one day. At ~7.8 KB resident per series:

~11M series x ~7.8 KB ≈ 85 GB of RAM, from one label on one metric.

That is an estimate (measured per-series cost x measured distinct count), and for context it is about 12x the entire ~7 GB that whole Prometheus uses. That label would not make Prometheus expensive; it would make it OOM.

InfluxDB has the same trap. The same setup ran both engines, and InfluxDB’s cost scales with series cardinality just as Prometheus’s does: the index it has to hold grows with the number of distinct tag-set combinations, not with how many points you write.

Measured on InfluxDB OSS 1.12: ~600k active series across all databases (SHOW SERIES CARDINALITY; the telegraf database alone is nearly all of it, across about a hundred measurements and tens of GB on disk), the same order of magnitude as the Prometheus above. And it costs RAM to match: influxd process RSS about 7 GB (/proc/<pid>/status VmRSS), with a peak (VmHWM) of about 19 GB.

Now the two engines are measured on the same basis, process resident memory, so the comparison is like for like:

Both RSS figures come from the OS: Prometheus process_resident_memory_bytes, InfluxDB /proc VmRSS. The exact per-series figure differs by engine, storage format, and tag layout; the direction never does. Both hold roughly 600k to 900k series in ~7 GB of RAM. The bill comes from cardinality, not from the datapoint rate.

A caveat on how InfluxDB pays, because the shape differs from Prometheus. RAM-scaling-with-cardinality is the documented, central design property of InfluxDB 1.x, not an anomaly. From the InfluxDB v1 TSI docs: the original engine kept the entire series index in memory, which “places an upper bound on the number of series a machine can hold … usually somewhere between 1 - 4 million series.” The Time Series Index (TSI) was built specifically to escape that ceiling by moving the index to disk (memory-mapped index files + OS page cache for hot data). InfluxData wrote a whole second index engine because cardinality-drives-RAM was a hard wall.

That disk-backed, memory-mapped design is why InfluxDB’s RSS is not the clean “committed heap” Prometheus’s is, and the split is measurable. Breaking influxd resident memory into its anonymous (truly owned) vs file-backed (mmap’d, reclaimable) parts, from /proc/<pid>/smaps_rollup and status: RSS about 7.2 GB = RssAnon about 2.5 GB (Go heap + off-heap allocations, hard-committed, lost on OOM) + RssFile about 4.7 GB (memory-mapped TSI index). And Private_Clean equals the whole file-backed part, i.e. all of the file-backed RSS is clean pages the kernel can evict for free and re-read from the index files on disk. Only ~35% of the footprint is committed; ~65% is reclaimable page cache.

That reframes two things. First, the ~19 GB VmHWM was mostly file-backed index paging; influxd would not have OOM’d there, because the kernel reclaims clean file pages first. Second, the per-series figure depends on which memory you count:

On the committed basis InfluxDB is lighter per series, because it pushes most of its index to disk via mmap, where Prometheus keeps its index in the heap (all anonymous, all committed). That is the TSI design working as intended. (The 10M-series user who saw RSS >200 GB, influxdb#23449, was watching the same mmap’d index, off-heap by design, dominate RSS.)

Net: both engines make you pay for cardinality, but in different currencies: Prometheus in committed heap (you provision RAM or it OOMs), InfluxDB largely in a disk-backed index paged into RAM (you provision RAM and disk, and the page cache absorbs some spikes). Neither forgives an unbounded label the way Elasticsearch’s inverted index does.

And the cardinality there is driven by exactly the tag-explosion pattern this section is about, a high-cardinality tag attached to a measurement (SHOW SERIES CARDINALITY FROM /.*/):

A single measurement with one badly chosen tag reaches hundreds of thousands of jars. The InfluxDB tag is the exact analog of the Prometheus label, and the same rule applies to it.

Contrast the same field in Elasticsearch. Those ~11M distinct request lines already live there: they are indexed terms inside the ~25M documents that cost about 12 GB primary on disk (about 24 GB with the replica). The millionth distinct value costs the same to index as the first; the inverted index was built to hold many unique terms. You pay disk, not RAM-resident series, and the cluster does not care.

Measured on real traffic: Elasticsearch is more expensive, but it forgives what Prometheus does not. The rest of this article asks whether you need that forgiveness. For an aggregated metric, you do not, and paying Elasticsearch storage to buy cardinality-tolerance you will never use is a rational choice at the wrong price.

The one rule to leave with

Before making a field a metric label, ask: “Do I need to aggregate or alert on this dimension, AND does its shelf stop growing?” Both yes: label it. Shelf never stops growing: it is a log or a trace, not a metric.

The hybrid, done right

The previous section is a “do not.” This one is the “do”, and it is the pattern almost every team wants, because it answers both questions at once: how often is this happening (aggregate) and which exact requests were they (instances). You do not choose one store. You use each for what it is cheap at, and you wire a jump between them.

The shape is one bounded metric plus the log store behind it:

http_requests_total{route, status, method}      # the metric: all labels bounded
                                                 # lives in Prometheus/InfluxDB
... and every individual request, with its full path, client_ip, trace,
    user_agent, timing ...                       # lives in Elasticsearch

Every label on the metric is bounded: status (25), method (9), and route. This works because route is the normalised template, not the raw path. Raw uri_path was the ~7M-distinct bomb from the last section; route is /api/:v/products/:id, not /api/0.6/products/12345.

Does the metric stay bounded? (measured)

Teams usually worry at this point that the route label will explode like the path did. On real traffic it does not. Measured in Elasticsearch on edge-logs-2026.06.29, API traffic only (about a quarter of all requests that day):

• raw uri_path: ~150k distinct, unusable as a label
• ~100 normalised route templates cover 90.7% of all API traffic (top-100 buckets of a path-normalising terms agg / total)

So the realistic series budget for the metric is:

~100 routes x 25 statuses x 9 methods = ~22,500 series ceiling, and in practice far fewer, because most route/status/method combinations never occur (DELETE on a read-only route, 502 on an endpoint that never 502s). That is one metric with low tens of thousands of series, fully alertable. Compare the ~11M series the raw path would have cost.

A caveat I learned doing this: normalisation is not a regex you bolt on after the fact. A naive “collapse version + trailing digits” pass over the real paths still left ~120k templates (free-text slugs and search terms hide inside API paths). The clean ~100-route result is the top of the distribution; getting a clean label end-to-end means the application emits its route template (the matched router pattern), not a post-hoc guess at the proxy. If you only have raw paths, you get the top-N concentration, not a truly bounded label.

The jump: from a metric spike to the exact requests

The metric tells you that /api/:v/checkout/order started throwing 500s at 14:05. It can never tell you which orders, for which customers, with what payload; that would be the unbounded detail you correctly kept out of the metric. That detail is one query away, in the log store.

Step 1, in Prometheus, you see the spike (the metric earns its keep here, cheaply):

sum by (route) (
  rate(http_requests_total{status="500", route="/api/:v/checkout/order"}[5m])
)

Step 2, you jump to Elasticsearch for the specific instances behind that spike, pivoting on the same bounded fields plus the time window the metric pointed at:

POST edge-logs-2026.06.29/_search
{
  "size": 50,
  "sort": [{ "@timestamp": "desc" }],
  "query": { "bool": { "filter": [
    { "term":  { "http_status_code": 500 } },
    { "term":  { "http_method.keyword": "POST" } },
    { "prefix":{ "uri_path.keyword": "/api/" } },
    { "wildcard": { "uri_path.keyword": "*/checkout/order*" } },
    { "range": { "@timestamp": { "gte": "2026-06-29T14:05:00Z",
                                 "lte": "2026-06-29T14:10:00Z" } } }
  ] } },
  "_source": ["@timestamp","uri_path","client_ip","user_agent",
              "response_time","backend_name","server_name"]
}

Now you have the full, high-cardinality detail (exact path with its IDs, the client, the backend that served it, the response time) for those 500s and no others. The cardinality you banned from the metric is exactly what Elasticsearch is cheap at storing and querying: it is an indexed term, not a resident series.

The verdict, both stores

• In the TSDB, http_requests_total{route,status,method} has ~100 routes and ~tens of thousands of series at most. It is cheap to keep resident and cheap to alert on, and the shelf is bounded by construction.
• In Elasticsearch, every request is a document, full path, client_ip, user_agent, timing, ~25M docs/day, ~12 GB primary, and the query above pulls the exact instances in milliseconds because every field is an indexed term.

The metric answers how much and is it broken. The log store answers which ones and why. The link between them is the set of bounded labels they share, route, status, method, and the timestamp. Put the bounded dimensions on both sides so you can pivot; keep the unbounded detail on the Elasticsearch side only. That hybrid is the default answer rather than a compromise.

The two bills

Everything above is mechanism. This section puts prices on it, because the argument hangs on one cost reality and that needs real numbers rather than adjectives.

The methodology matters, so I state it up front: I do not compare “1M logs vs 1M metrics.” That is meaningless; a log document and a metric datapoint differ by two orders of magnitude in size and compressibility. The useful comparison is a scenario: the edge takes ~25M requests a day; what does each store cost to answer its half of the question?

Both columns below are the same day of the same traffic, priced two ways.

How the table was built. Metric side, from the Prometheus self-metrics: scrape interval ~36 s implied from head_series / rate(head_samples_appended_total), on-disk ~1.3 bytes/sample (~126 B/chunk over ~98 samples/chunk), ~7.8 KB per series in RAM. Series budget from the hybrid section (~100 routes, ~3k combinations actually occurring, ~22.5k ceiling). Log side: _cat/indices for the daily index, primary plus one replica, best_compression, ~500 bytes/doc compressed; the retention row is the sum over all retained daily indices.

Read the storage row again. For the aggregate question, is checkout throwing 5xx, and at what rate, the metric is roughly 180x to 1,300x cheaper than the logs, depending on how many route/status/method combinations actually occur. A gap that size is the difference between a metric you forget you are paying for and a log stream you budget clusters around.

And note what you are not getting for the Elasticsearch bill: the ~24 GB/day does not buy a lower storage bill for the aggregate you already have a metric for, and it does not buy a way out of paying for that aggregate’s computation on every ask. (Which side wins on raw compute time is not fixed. “The materialization check” earlier measures it both ways: avg-style aggregates are expensive query-time in Elasticsearch, count-style aggregates were actually cheaper in Elasticsearch than the equivalent Prometheus increase() on the same traffic. What is fixed is that write-time pays once, at ingest, and query-time pays again on every ask, regardless of which side happens to be faster for a given aggregate.) The Elasticsearch bill buys forgiveness: the ability to store the full request line, the client IP, the user agent, the per-request timing, all unbounded, and pull any slice of it back later. You pay ~500 bytes per document, 25 million times a day, for the right to ask which ones after the fact.

The thesis, then: Elasticsearch is more expensive, but it forgives what the TSDB does not. For the aggregate, you do not need the forgiveness, so you keep it on the metric for ~180x less. For the specific instance, the forgiveness is the entire point, so you pay Elasticsearch and it is worth it. The mistake I want to prevent is paying the Elasticsearch bill, or worse the Prometheus OOM, for the aggregate, by putting an unbounded field where a bounded label belonged.

A fair word about Loki

Everything above pits a TSDB against Elasticsearch, because that is the choice in front of most teams I have worked with. But “log store” does not have to mean “full Elasticsearch,” and I do not want to leave the impression that Kibana is simply the better log tool. It is better at one specific thing and worse at another.

The difference is what gets indexed. Elasticsearch indexes the content of every document: every field, every term, full-text. That is what makes the ~24 GB/day bill, and it is also what lets you search user_agent for a substring or aggregate a free-text field after the fact. Loki does the opposite: it indexes only a small set of labels and stores the log body as an opaque, compressed blob. You cannot do a rich content aggregation over a Loki stream, but you also do not pay to index text you were only ever going to grep.

So the split is by access pattern, not by product quality:

• Full-text search, or heavy aggregation over log content (analytics on the request body, faceting on arbitrary fields, the jump-to-instance query from the hybrid section): that is Elasticsearch/Kibana. The inverted index is the whole point and it earns its cost.
• Grep-style logs sitting next to metrics on one dashboard, at minimal storage cost (tail this service, filter by a couple of labels, correlate with a Grafana panel): that is Loki. It was built precisely to store logs cheaply by indexing labels only.

The thesis holds for Loki too. The cardinality rule applies to Loki labels exactly as it applies to metric labels: a high-cardinality Loki label (a trace_id as a label rather than in the body) is the same bomb in a different housing. The choice is never “metrics good, logs bad.” Match the store to the access pattern, and keep unbounded identifiers out of whatever the store indexes.