Denys Miasnychenko Software Architect / DevOps
Nav	About Resume ES - Guides Articles

One node or three, and growing from one to three

Verified against Elasticsearch 9.5.5 only, on 29 October 2026; OpenSearch 3.9.0 was not tested in the lab. Configs: github.com/jtsunne/small-elasticsearch-lab at v0.1.0.

One node is fine when you can rebuild or restore its data within the time you can afford to be down. Three master-eligible nodes is the smallest cluster that keeps accepting writes when a node dies. Two nodes buys you little: one of them can still stop the cluster. To get from one to three, build a new cluster and restore a snapshot into it; growing the existing node in place also worked in my lab, once, but it touches the only copy of your data.

Topology	Keeps a master after losing	Writes during that loss	Basis
One node	nothing	none until the node is back; snapshots are the only redundancy	[official]
Two nodes, both master-eligible	only the node that is not the single voter	the voter’s loss stops the cluster	[official]
Two nodes, one master-eligible	the other node	losing the master-eligible node stops the cluster	[official]
Two nodes plus a voting-only tiebreaker	any one node	accepted	[official], not tested
Three master-eligible nodes	any one node	accepted (L-T1)	[measured]
Three nodes, two lost	nothing	reads from the last node’s local copy; writes held for 60 s, then 503 (L-T2)	[measured]

What I ran: es-three-local (30 of 30 rows, plus fault-test.sh 15 of 15), growth procedure A es-grow-restore (22 of 22 end-state rows) and procedure B es-grow-inplace (22 of 22), with the traps L-G3 and L-G4, on 29 October 2026, on my laptop (Linux, cgroup v2, 12 logical CPUs, about 15.5 GiB of RAM, Docker 29.9.1, Compose 5.6.0), with Elasticsearch 9.5.5 and Caddy 2.11.7. Three nodes on one laptop share one CPU, one disk and one kernel, so this shows how quorum and allocation behave, not fault tolerance. Each procedure ran once, with 1,001 documents: the times below measure the mechanics, not a data volume. Nothing on this page was tested on OpenSearch. Row ids and TAP files are in results/tests/ of the companion repo.

Quorum in one paragraph

Elasticsearch elects a master from the master-eligible nodes in the voting configuration, and needs a majority of that configuration to do it. With an even number of master-eligible nodes, “Elasticsearch leaves one of them out of the voting configuration to ensure that it has an odd size”, which “does not decrease the failure-tolerance of the cluster”. New master-eligible nodes join the voting configuration by themselves, and cluster.auto_shrink_voting_configuration (default true) keeps it at three or more while three are available. The rule to remember: “you must not stop half or more of the nodes in the voting configuration at the same time.” [official] The lab saw the odd-size rule at work: the voting configuration had 1 member after the second node joined, and 3 after the third (procedure B, step 3).

One node

Replicas must be 0 for the node to be green, because a replica is never placed on the same node as its primary. Snapshots are then the only copy of the data that survives the node. That is acceptable when the data can be re-ingested from its source, or restored from a snapshot, within the downtime you can afford. The previous page builds this node.

Two nodes

With two master-eligible nodes the voting configuration holds only one of them. Lose that one and there is no master. Elastic’s advice for two nodes: “We recommend you set only one of your two nodes to be master-eligible”, and “we do not recommend deploying a two-node cluster in production”. [official] You get a replica, which protects the data, but not availability: one specific node still stops the cluster. Not tested in the lab.

Two plus a tiebreaker

A third, small node that only votes: a “dedicated voting-only master-eligible node, also known as a dedicated tiebreaker”, and “the two original nodes should not be voting-only”. [official] It holds no data, so it can be much smaller than the data nodes. I have not tested this topology and have no memory figure for the tiebreaker.

Three nodes

All three master-eligible and data, one replica. Any one node can fail. The cost at this scale is plain: three times the memory and disk of one node, and each node still has the same small heap. In the lab each node had cpus: 1, mem_limit: 2g and 1 GiB of heap (es-three-local-09), so the host needed about 7 GiB free: 3 x 2 GiB for the container limits plus about 1 GiB for Caddy and the host [estimate], and every index needed twice its size on disk.

The quorum simulator below applies the documented voting and allocation rules to a topology you choose. It is a model, not Elasticsearch.

Elasticsearch presets, one index with 1 primary shard; worst and best choice of the failed nodes
Preset	Healthy	Lose one node	Lose two at once	Lose two, one at a time
1 node, 0 replicas	Master: es01. Health: GREEN. Writes: accepted.	all nodes down	not enough nodes	not enough nodes
2 nodes, both master-eligible, 1 replica	Master: es01. Health: GREEN. Writes: accepted.	worst: no master, no health, writes rejected [uncertain] (lose es01); best: master elected, YELLOW, writes accepted [uncertain] (lose es02)	all nodes down [uncertain]	all nodes down [uncertain]
2 nodes, one master-eligible, 1 replica	Master: es01. Health: GREEN. Writes: accepted.	worst: no master, no health, writes rejected [uncertain] (lose es01); best: master elected, YELLOW, writes accepted (lose es02)	all nodes down	all nodes down
2 nodes + voting-only tiebreaker, 1 replica	Master: es01. Health: GREEN. Writes: accepted.	worst: master elected, YELLOW, writes accepted (lose es01); best: master elected, GREEN, writes accepted (lose es03)	no master, no health, writes rejected [uncertain]	no master, no health, writes rejected [uncertain]
3 nodes, 1 replica	Master: es01. Health: GREEN. Writes: accepted.	master elected, GREEN, writes accepted	no master, no health, writes rejected [uncertain]	no master, no health, writes rejected [uncertain]
5 nodes, 1 replica	Master: es01. Health: GREEN. Writes: accepted.	master elected, GREEN, writes accepted	worst: master elected, RED, writes rejected (lose es01 and es02); best: master elected, GREEN, writes accepted (lose es04 and es05)	master elected, GREEN, writes accepted

[model] The docs do not say which node is left out. The model leaves out the highest id, so which single node loss stops a two-node cluster is a model choice: in reality it can be either.
[uncertain] Without a master the model reports no health: the cluster health API depends on the elected master (from training, not stated by the cited pages).

OpenSearch 3.9.0: the same presets

OpenSearch says cluster manager for master and has no voting-only role, so there is no tiebreaker preset.

OpenSearch presets, one index with 1 primary shard
Preset	Healthy	Lose one node	Lose two at once	Lose two, one at a time
1 node, 0 replicas	Cluster manager: os01. Health: GREEN. Writes: accepted.	all nodes down	not enough nodes	not enough nodes
2 nodes, both cluster-manager-eligible, 1 replica	Cluster manager: os01. Health: GREEN. Writes: accepted.	worst: no cluster manager, no health, writes rejected [uncertain] (lose os01); best: cluster manager elected, YELLOW, writes accepted [uncertain] (lose os02)	all nodes down [uncertain]	all nodes down [uncertain]
2 nodes, one cluster-manager-eligible, 1 replica	Cluster manager: os01. Health: GREEN. Writes: accepted.	worst: no cluster manager, no health, writes rejected [uncertain] (lose os01); best: cluster manager elected, YELLOW, writes accepted (lose os02)	all nodes down	all nodes down
3 nodes, 1 replica	Cluster manager: os01. Health: GREEN. Writes: accepted.	cluster manager elected, GREEN, writes accepted	no cluster manager, no health, writes rejected [uncertain]	no cluster manager, no health, writes rejected [uncertain]
5 nodes, 1 replica	Cluster manager: os01. Health: GREEN. Writes: accepted.	cluster manager elected, GREEN, writes accepted	worst: cluster manager elected, RED, writes rejected (lose os01 and os02); best: cluster manager elected, GREEN, writes accepted (lose os04 and os05)	cluster manager elected, GREEN, writes accepted

[model] The docs do not say which node is left out. The model leaves out the highest id, so which single node loss stops a two-node cluster is a model choice: in reality it can be either.
[uncertain] Without a master the model reports no health: the cluster health API depends on the elected master (from training, not stated by the cited pages).
[uncertain] OpenSearch documents cluster.no_cluster_manager_block with values all and write, but the page does not state the default; the model assumes write, as in Elasticsearch.

What the lab saw, Elasticsearch 9.5.5, three nodes on one host, run of 29 October 2026 [measured]:

L-T1, the elected master stops: a new master at the first poll, searches and a bulk write through Caddy accepted, health yellow and green again on two nodes 48 to 50 s later, timed from after those checks rather than from the stop. The model skips that delay and shows green at once.
L-T2, a second node stops: no master within a second. Searches still answered from the last node's local shard copy, possibly stale; a bulk write was held for about 60 s and then rejected with 503 cluster_block_exception (no master). A client with a 30 s timeout saw only a timeout, never the reason.

Adding an empty node to a formed Elasticsearch cluster
Existing node has `discovery.type: single-node`	New node has `cluster.initial_master_nodes`	Outcome	Basis
no	no	The new node joins the existing cluster: one cluster, one cluster UUID.	L-G2 [measured]
no	yes	The new node elects itself and forms a second cluster with its own cluster UUID under the same cluster name. Behind one proxy, writes would split between the two.	L-G3 [measured]
yes	no	The existing node refuses every join ("cannot join node with [discovery.type] set to [single-node] discovery") and logs that it is a fully-formed single-node cluster. Nothing splits; the cluster never grows.	L-G4 [measured]
yes	yes	Not run in the lab. Both traps apply: the new node forms its own cluster, and the existing node would refuse it anyway.	L-G3 and L-G4 [not measured]

What happened when nodes stopped

fault-test.sh stops the elected master, then a second node, and checks what still works through Caddy. Caddy has the three nodes as upstreams, round robin, with a retry when a connection to one fails.

Step	Observed	Evidence
Stop the elected master (L-T1)	a new master at the first poll after `docker compose stop` returned (0 s); yellow with 2 nodes and 3 or 4 unassigned shards	fault TAP check 4
Six searches through Caddy	all 200	check 5
Bulk create through Caddy	accepted	check 6
Back to green on two nodes	48 to 50 s in three runs, counted from after the searches and the write, not from the stop	check 7
Stop a second node (L-T2)	the last node reports no master within a second; `_cluster/health` answers `503 master_not_discovered_exception`	checks 8, 9
Search through Caddy	200 with hits, from the last node’s local shard copy (`_shards.total` 1)	check 9 diagnostics
Bulk create through Caddy	held for 60 s, then `503 cluster_block_exception`	check 10
Start both nodes again	green with 3 nodes 48 to 50 s after the start; the L-T1 write present, the L-T2 write absent, one `cluster_uuid` on all three	checks 11 to 15

The no-master write, as the client received it:

503 cluster_block_exception: blocked by: [SERVICE_UNAVAILABLE/2/no master];

What to take from this:

Losing one of three is a non-event for clients. The searches did not fail. That Caddy retried on another node is the likely reason; I did not record Caddy’s logs, so that part is a guess.
The time to green after a loss was not measured from the loss. The 48 to 50 s clock starts after the six searches and the write. It is consistent with the delay before Elasticsearch rebuilds replicas of a node that left, index.unassigned.node_left.delayed_timeout, which “defaults to 1m”, [official] but it does not measure that delay.
Without a master, reads keep working and writes hang. Reads may return stale data: cluster.no_master_block defaults to write, and Elastic says reads then “succeed, based on the last known cluster configuration”. [official] The write took 60.0 s to fail. With a 30 s client timeout the client saw nothing at all: curl exit 28, status 000, 0 bytes. An application with a timeout under a minute only ever sees timeouts, never the reason.

Readiness after start

docker compose up --wait reported all three nodes healthy while the .security-7 primary was still unassigned. The first setup call as elastic then failed:

503 unavailable_shards_exception: at least one primary shard for the index [.security-7] is unavailable
authentication_processing_error: failed to retrieve password hash for reserved user [elastic]

In my first run the setup script did not notice, and 25 rows failed for lack of an app key. Every script in the companion repo now waits for health and retries 503, 401, 408 and no answer for up to 180 seconds (300 in the slow growth steps). Do the same in anything that runs right after a start or restart.

Growing: procedure A, a new cluster plus a restore

Recommended. The old node keeps running and is never reconfigured, a new three-node cluster is built next to it, a snapshot of the old node is restored into it, and Caddy is switched over. Rollback is pointing Caddy back.

Register a shared filesystem repository on the old node. The directory must be mounted “to the same location on all master and data nodes” and listed in path.repo, which on a running node means a restart. [official] In the lab the old node, es00, started with path.repo set.
Take a rehearsal snapshot of app-* plus the security feature state, while the old node keeps taking writes. The feature state carries users, roles and API keys, so the application’s existing key keeps working after cutover.
Start the new cluster: three nodes, transport TLS from the first start, bootstrapped once with cluster.initial_master_nodes, a different cluster name. Register the same repository there read-only: “only one cluster should have write access to the repository”. [official]
Restore the rehearsal snapshot and compare document counts.
Block writes on the old node’s app-*, take the final snapshot, restore it with --replace on the new cluster. From the write block on, application writes fail until cutover.
Hand the repository over (unregister on the old node first, then register writable on the new one) and point Caddy at the three new nodes.

The restore request, with replicas set and the write block lifted on the way in:

POST /_snapshot/grow/<snapshot>/_restore?wait_for_completion=true
{
  "indices": "app-*",
  "include_global_state": false,
  "feature_states": ["security"],
  "index_settings": {"index.number_of_replicas": 1, "index.blocks.write": false}
}

What the run showed (results/tests/2026-10-29-es-grow-restore-steps.log, results/tests/2026-10-29-es-grow-restore.tap):

Step	Observed
Before cutover	two clusters with different `cluster_uuid`, as designed
Snapshots	both `SUCCESS`, with `.security-7`, `app-a`, `app-b` and the `security` feature state
Restore plus replica recovery	5 s for the rehearsal, 5 s for the final [measured]
Document counts	equal after both restores (`app-a` 1, `app-b` 1000)
Restored indices	1 replica, `index.blocks.write` false: `index_settings` overrides the block stored in the final snapshot (`es-grow-restore-03`, `-06`)
The API key created on the old node	authenticated on the new cluster (`es-grow-restore-10`)
Caddy recreate	back after 2 s [measured]

Not measured: the write-blocked window from the final snapshot to cutover, which is the real downtime of A for writes, and client errors during the Caddy recreate. Not run: the rollback script. On a real host, reload Caddy instead of recreating it; caddy reload swaps the config without closing the listener.

Growing: procedure B, in place

Rehearsed once in my lab, not an Elastic-published procedure. No Elastic page describes turning a node started with discovery.type: single-node into the first node of a cluster; the steps follow the bootstrapping and add-node pages and the Elasticsearch source. Prefer A. B changes the only node that holds the data, and once a second node has joined there is no in-place way back.

Preflight, nothing changes yet: vm.max_map_count set to 1048576 [official], transport certificates for all nodes from elasticsearch-certutil, the node’s cluster_uuid recorded, and a snapshot taken. That snapshot is the only way back after step 3.
Restart es01 with the same data volume, without discovery.type: single-node, with discovery.seed_hosts, transport TLS and roles master,data,ingest. Never set cluster.initial_master_nodes: the cluster was bootstrapped once already, by es01 in single-node mode. “Do not configure this setting on nodes joining an existing cluster. Do not configure this setting on nodes which are restarting.” [official] From here es01 runs in production mode and the bootstrap checks apply.
Start es02, then es03, each with an empty data path, and check that each reports es01’s cluster_uuid.
Set replicas to 1 and wait for green. Between step 3 and green each shard has one copy; losing a node in that window loses data.
Point Caddy at all three nodes, only after all three report the same cluster_uuid.

The state after growth, the file the steps converge on:

# Procedure B, state after growth: es01 keeps its data and cluster UUID, loses
# discovery.type single-node and gains transport TLS; es02 and es03 join it with
# empty data paths. No node sets cluster.initial_master_nodes: the cluster was
# bootstrapped once, by es01 in single-node mode, and must never bootstrap
# again. Start from step-0-seed.sh, not from this file. Lab only: three nodes
# on one host.
name: es-grow-inplace

x-cluster-env: &cluster-env
  cluster.name: es-grow-inplace
  node.roles: master,data,ingest
  discovery.seed_hosts: es01,es02,es03
  path.repo: /usr/share/elasticsearch/snapshots
  xpack.security.enabled: "true"
  xpack.security.http.ssl.enabled: "false"
  xpack.security.transport.ssl.enabled: "true"
  xpack.security.transport.ssl.verification_mode: certificate
  xpack.security.transport.ssl.key: certs/node-key.pem
  xpack.security.transport.ssl.certificate: certs/node.pem
  xpack.security.transport.ssl.certificate_authorities: certs/ca.pem
  xpack.license.self_generated.type: basic
  bootstrap.memory_lock: "true"
  ELASTIC_PASSWORD: ${ELASTIC_PASSWORD:?set ELASTIC_PASSWORD in .env}
  ES_JAVA_OPTS: ${ES_HEAP_OPTS--Xms1g -Xmx1g}

x-es-node: &es-node
  image: ${ES_IMAGE:?ES_IMAGE comes from versions.env}
  cpus: 1
  mem_limit: 2g
  ulimits:
    memlock:
      soft: -1
      hard: -1
    nofile:
      soft: 65535
      hard: 65535
  networks:
    - backend
  healthcheck:
    test: ["CMD-SHELL", "curl -s http://localhost:9200 | grep -q 'missing authentication credentials'"]
    interval: 10s
    timeout: 10s
    retries: 30
    start_period: 30s

x-snapshots: &snapshots
  type: bind
  source: ${SNAPSHOT_DIR:?set SNAPSHOT_DIR in .env}
  target: /usr/share/elasticsearch/snapshots
  bind:
    create_host_path: false

x-ca: &ca
  type: bind
  source: ./certs/ca.pem
  target: /usr/share/elasticsearch/config/certs/ca.pem
  read_only: true
  bind:
    create_host_path: false

services:
  es01:
    <<: *es-node
    environment:
      <<: *cluster-env
      node.name: es01
    volumes:
      - esdata01:/usr/share/elasticsearch/data
      - *snapshots
      - *ca
      - {type: bind, source: ./certs/es01.pem, target: /usr/share/elasticsearch/config/certs/node.pem, read_only: true, bind: {create_host_path: false}}
      - {type: bind, source: ./certs/es01-key.pem, target: /usr/share/elasticsearch/config/certs/node-key.pem, read_only: true, bind: {create_host_path: false}}

  es02:
    <<: *es-node
    environment:
      <<: *cluster-env
      node.name: es02
    volumes:
      - esdata02:/usr/share/elasticsearch/data
      - *snapshots
      - *ca
      - {type: bind, source: ./certs/es02.pem, target: /usr/share/elasticsearch/config/certs/node.pem, read_only: true, bind: {create_host_path: false}}
      - {type: bind, source: ./certs/es02-key.pem, target: /usr/share/elasticsearch/config/certs/node-key.pem, read_only: true, bind: {create_host_path: false}}

  es03:
    <<: *es-node
    environment:
      <<: *cluster-env
      node.name: es03
    volumes:
      - esdata03:/usr/share/elasticsearch/data
      - *snapshots
      - *ca
      - {type: bind, source: ./certs/es03.pem, target: /usr/share/elasticsearch/config/certs/node.pem, read_only: true, bind: {create_host_path: false}}
      - {type: bind, source: ./certs/es03-key.pem, target: /usr/share/elasticsearch/config/certs/node-key.pem, read_only: true, bind: {create_host_path: false}}

  caddy:
    image: ${CADDY_IMAGE:?CADDY_IMAGE comes from versions.env}
    depends_on:
      es01:
        condition: service_healthy
      es02:
        condition: service_healthy
      es03:
        condition: service_healthy
    environment:
      - ES_SITE_ADDRESS=${ES_SITE_ADDRESS:?set ES_SITE_ADDRESS in .env}
      - ALLOWED_CIDRS=${ALLOWED_CIDRS:?set ALLOWED_CIDRS in .env}
      - ES_UPSTREAMS=es01:9200 es02:9200 es03:9200
    ports:
      - "${CADDY_BIND:-127.0.0.1}:${CADDY_HTTP_PORT:-80}:80"
      - "${CADDY_BIND:-127.0.0.1}:${CADDY_HTTPS_PORT:-443}:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    networks:
      - backend

networks:
  backend: {}

volumes:
  esdata01: {}
  esdata02: {}
  esdata03: {}
  caddy_data: {}
  caddy_config: {}

Tested at v0.1.0: scenarios/es-grow-inplace/compose.yaml.

What the run showed (results/tests/2026-10-29-es-grow-inplace-steps.log, results/tests/2026-10-29-es-grow-inplace.tap):

Step	Observed
es01 restarted out of single-node mode	`cluster_uuid` unchanged, voting configuration 1
es01 downtime	answered again 55 s after the stop [measured]; through Caddy, requests fail for that long
es02 joined	voting configuration 1
es03 joined	voting configuration 3
Replicas set to 1	green after 2 s [measured]
Caddy recreate	2 s [measured]
End state	same `cluster_uuid` on all three nodes, counts equal to step 1, the API key from before the growth still valid, each `app-*` shard on two nodes (`es-grow-inplace-08`, `-11`)

Not run: the rollback script, client traffic during the restart, losing a node between steps 3 and 4, and a host where the bootstrap checks of step 2 fail.

Traps, with the log lines

L-G3: cluster.initial_master_nodes on a joining node gives you a second cluster. I started es04 with cluster.initial_master_nodes=es04 and the seed hosts of the grown cluster. It logged:

This node has not joined a bootstrapped cluster yet; [cluster.initial_master_nodes] is set to [es04]

elected itself about half a second later, and formed a one-node cluster with its own cluster_uuid, under the same cluster name. es01 kept listing es01 to es03. Thirty seconds later es04 logged the “fully-formed single-node cluster” warning quoted below. The docs call this a risk; here it happened on the first try. Why es04 did not join the existing master first is my guess, not a checked fact: with itself as the only initial master node, it has a quorum of one before discovery reaches the others. With Caddy in front of both clusters, writes would split between them, and Elasticsearch never merges clusters. es04 got no traffic in the lab.

L-G4: a leftover discovery.type: single-node. I recreated es01 with discovery.type: single-node on top of the seed hosts and transport TLS. It started, answered after 48 s, listed only itself, and logged every 30 s:

This node is a fully-formed single-node cluster with cluster UUID [ySZebGm0SemWUgu2fpt5Vg], but it is configured as if to discover other nodes and form a multi-node cluster via the [discovery.seed_hosts=[es01, es02, es03]] setting. Fully-formed clusters do not attempt to discover other nodes, and nodes with different cluster UUIDs cannot belong to the same cluster. The cluster UUID persists across restarts and can only be changed by deleting the contents of the node's data path(s). Remove the discovery configuration to suppress this message. See [https://www.elastic.co/docs/deploy-manage/distributed-architecture/discovery-cluster-formation/modules-discovery-bootstrap-cluster?version=9.5#modules-discovery-bootstrap-cluster-joining] for more information.

es02, with an empty data path, tried to join about once a second for 60 s. All 19 logged attempts failed with es01’s rejection:

java.lang.IllegalStateException: cannot join node with [discovery.type] set to [single-node] discovery

This one does not split anything. The cluster simply never grows, and the log says why.

Two more, from the procedure scripts rather than the traps:

A stale “way back” snapshot. The snapshot directory is a bind mount, and docker compose down -v does not delete it. On a second run, step 1 printed snapshot before-grow already exists, keeping it: the kept snapshot came from the previous run’s cluster. Check the date of the snapshot you are relying on before step 2.
Copying a data directory to a new node. The copy carries the old node’s identity and cluster UUID. A node with another cluster’s data path is rejected at join with “This node previously joined a cluster with UUID […] and is now trying to join a different cluster with UUID […]” (from the Elasticsearch source; not triggered in the lab). New nodes start empty; the cluster copies shards to them.

Generate the files

The config generator writes the files of a scenario: one node, one node with Kibana or Dashboards, or the three-node lab cluster (Elasticsearch only). For the inputs the lab ran, its output is the tested files, apart from a short generated header. Changed values such as memory or host names keep the tested structure and are listed in the file header; a structural change, such as a disabled route or an IPv6 allowlist, marks the output as not tested. Without JavaScript it shows the tested single-node files.

Set vm.max_map_count=1048576 on the host; a container cannot set it.

.env.example

# Copy to .env, or run ./init-env.sh, which writes .env with random passwords.
# Passwords: alphanumeric, at least 6 characters (Elasticsearch minimum); the
# scripts here refuse the change-me placeholders.
ELASTIC_PASSWORD=change-me-elastic
APP_PASSWORD=change-me-app

# Lab defaults: Caddy on loopback, high ports, Caddy's internal CA for *.localhost.
# On a real host: CADDY_BIND=0.0.0.0, ports 80 and 443, ES_SITE_ADDRESS set to a
# public DNS name so Caddy obtains a certificate.
CADDY_BIND=127.0.0.1
CADDY_HTTP_PORT=8080
CADDY_HTTPS_PORT=8443
ES_SITE_ADDRESS=es.localhost

# Space-separated CIDRs. Caddy sees the immediate TCP peer, see README.md.
ALLOWED_CIDRS=127.0.0.1/32

# Unset: -Xms1g -Xmx1g. Empty string: Elasticsearch sizes the heap itself.
#ES_HEAP_OPTS=

Caddyfile

# Requires Caddy 2.11.1 or later (CVE-2026-27587). The {$VAR:default} values let
# the same file run in compose, in CI validation and in the proxy-only test
# harness; compose always sets ES_SITE_ADDRESS and ALLOWED_CIDRS.
{
	admin {$CADDY_ADMIN:localhost:2019}
	servers {
		protocols h1 h2
	}
}

{$ES_SITE_ADDRESS:es.localhost} {
	log

	# route keeps the written order. Without it Caddy sorts directives and runs
	# handle and reverse_proxy before abort, so the IP allowlist would not
	# protect the proxied paths.
	route {
		# remote_ip is the immediate TCP peer. Behind another proxy or Docker's
		# userland proxy that is not the client, and this check means nothing.
		@outside not remote_ip {$ALLOWED_CIDRS:127.0.0.1/32}
		abort @outside

		request_body {
			max_size 20MB
		}

		# Matched against the raw request URI, because reverse_proxy forwards the
		# raw path: path and path_regexp see a decoded, dot-cleaned path, so
		# /secret/../app-a/_search would match them and still reach the node as
		# written. Any encoding, dot segment, double slash, upper case, comma,
		# wildcard, date math or remote-cluster prefix fails the pattern and
		# gets the 404 below.
		# This allowlist is a speed bump: a _bulk body can still name any index,
		# and only the Elasticsearch role decides what the caller may touch.
		@app_read {
			method GET POST
			expression `{http.request.uri}.matches("^/app-[a-z0-9][a-z0-9-]*/_(search|count)([?].*)?$")`
		}
		@app_write {
			method POST PUT
			expression `{http.request.uri}.matches("^/app-[a-z0-9][a-z0-9-]*/_bulk([?].*)?$")`
		}

		reverse_proxy @app_read {$ES_UPSTREAM:es01:9200} {
			# Elasticsearch does not honour this header; dropping it only keeps
			# the upstream request equal to what the method matcher saw.
			header_up -X-HTTP-Method-Override
		}
		reverse_proxy @app_write {$ES_UPSTREAM:es01:9200} {
			header_up -X-HTTP-Method-Override
		}

		respond 404
	}
}

compose.yaml

name: es-single

services:
  es01:
    image: ${ES_IMAGE:?ES_IMAGE comes from versions.env}
    environment:
      - node.name=es01
      - cluster.name=es-single
      - discovery.type=single-node
      - xpack.security.enabled=true
      # Allowed without node certificates only because discovery.type is
      # single-node; a second node needs transport TLS.
      - xpack.security.http.ssl.enabled=false
      - xpack.security.transport.ssl.enabled=false
      - xpack.license.self_generated.type=basic
      - bootstrap.memory_lock=true
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD:?set ELASTIC_PASSWORD in .env}
      # Same value auto-sizing picks for a 2 GiB container, set explicitly so the
      # OpenSearch scenarios (which do not auto-size) run the same heap. Set
      # ES_HEAP_OPTS to an empty string to let Elasticsearch size it (L-M1).
      - ES_JAVA_OPTS=${ES_HEAP_OPTS--Xms1g -Xmx1g}
    cpus: 1
    mem_limit: 2g
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65535
        hard: 65535
    volumes:
      - esdata:/usr/share/elasticsearch/data
    networks:
      - backend
    healthcheck:
      test: ["CMD-SHELL", "curl -s http://localhost:9200 | grep -q 'missing authentication credentials'"]
      interval: 10s
      timeout: 10s
      retries: 30
      start_period: 30s

  caddy:
    image: ${CADDY_IMAGE:?CADDY_IMAGE comes from versions.env}
    depends_on:
      es01:
        condition: service_healthy
    environment:
      - ES_SITE_ADDRESS=${ES_SITE_ADDRESS:?set ES_SITE_ADDRESS in .env}
      - ALLOWED_CIDRS=${ALLOWED_CIDRS:?set ALLOWED_CIDRS in .env}
    ports:
      - "${CADDY_BIND:-127.0.0.1}:${CADDY_HTTP_PORT:-80}:80"
      - "${CADDY_BIND:-127.0.0.1}:${CADDY_HTTPS_PORT:-443}:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    networks:
      - backend

networks:
  backend: {}

volumes:
  esdata: {}
  caddy_data: {}
  caddy_config: {}

init-env.sh

#!/usr/bin/env bash
# Writes .env next to this script from .env.example with random alphanumeric
# passwords. Refuses to overwrite an existing .env.
set -euo pipefail

dir=$(cd "$(dirname "$0")" && pwd)
env_file="$dir/.env"

if [ -e "$env_file" ]; then
	echo "error: $env_file exists; remove it first to regenerate" >&2
	exit 1
fi

random_secret() {
	LC_ALL=C tr -dc 'A-Za-z0-9' </dev/urandom | head -c "${1:-32}" || true
}

umask 077
while IFS= read -r line; do
	case "$line" in
	*=change-me*) printf '%s=%s\n' "${line%%=*}" "$(random_secret 32)" ;;
	*) printf '%s\n' "$line" ;;
	esac
done <"$dir/.env.example" >"$env_file"
echo "wrote $env_file (mode 600)"

setup-roles.sh

#!/usr/bin/env bash
# Creates the index template, the test indices, the app_writer role and the
# app_x user, then creates an expiring, scoped API key for the application as
# elastic. Run it again to rotate: it creates a new key, see README.md.
#
# Usage: setup-roles.sh [--key-out FILE]
# READY_TIMEOUT (seconds, default 180) bounds the wait for the cluster.
#
# The node publishes no port and Caddy does not proxy _security, so every call
# runs curl inside the es01 container against 127.0.0.1:9200. Passwords reach
# that curl on stdin and as a curl config read from a pipe, never through argv,
# which any user on the host can read with ps.
set -euo pipefail

dir=$(cd "$(dirname "$0")" && pwd)
repo=$(cd "$dir/../.." && pwd)
key_out=""

while [ $# -gt 0 ]; do
	case "$1" in
	--key-out) key_out="${2:?--key-out needs a file}"; shift 2 ;;
	-h | --help) sed -n '2,6p' "$0"; exit 0 ;;
	*) echo "error: unknown argument $1" >&2; exit 2 ;;
	esac
done

command -v jq >/dev/null || { echo "error: jq is required" >&2; exit 1; }
[ -f "$dir/.env" ] || { echo "error: $dir/.env missing, run init-env.sh" >&2; exit 1; }

# Read single keys instead of sourcing .env: ALLOWED_CIDRS holds spaces and
# compose does not quote values.
env_value() {
	local v
	v=$(grep -m1 "^$1=" "$dir/.env" | cut -d= -f2-) || true
	printf '%s' "${!1:-$v}"
}
ELASTIC_PASSWORD=$(env_value ELASTIC_PASSWORD)
APP_PASSWORD=$(env_value APP_PASSWORD)
for name in ELASTIC_PASSWORD APP_PASSWORD; do
	case "${!name}" in
	"" | change-me*) echo "error: $name is unset or still the .env.example placeholder" >&2; exit 1 ;;
	esac
done

compose=(docker compose --project-directory "$dir" --env-file "$repo/versions.env"
	--env-file "$dir/.env" -f "$dir/compose.yaml")

# The first stdin line is the password, the rest is the request body. bash
# reads a pipe byte by byte, so curl gets the stream right after that line.
# shellcheck disable=SC2016 # expanded inside the container
remote='IFS= read -r pass; pass=${pass//\\/\\\\}; pass=${pass//\"/\\\"}
curl -sS -w "\n%{http_code}" -X "$1" -H "Content-Type: application/json" \
	--data-binary @- -K <(printf "user = \"%s:%s\"\n" "$3" "$pass") \
	"http://127.0.0.1:9200$2"'

# es METHOD PATH < body; runs as elastic, prints the response body, then the
# status on its own line.
es() {
	{ printf '%s\n' "$ELASTIC_PASSWORD"; cat; } |
		"${compose[@]}" exec -T es01 bash -c "$remote" _ "$1" "$2" elastic
}

# es_ok METHOD PATH < body; prints the body, fails unless 2xx.
es_ok() {
	local res status body
	res=$(es "$@")
	status=${res##*$'\n'}
	body=${res%$'\n'*}
	case "$status" in
	2??) printf '%s' "$body" ;;
	*) echo "error: $1 $2 returned $status: $body" >&2; return 1 ;;
	esac
}

# wait_ready PATH: GET PATH until it answers 200. compose up --wait returns
# when the node answers HTTP, which can be before the .security-7 primary is
# assigned; until then even elastic fails with 503 ("failed to retrieve
# password hash for reserved user"), or 401. 401, 408 (health timeout), 503
# and no answer are retried until READY_TIMEOUT seconds (default 180) have
# passed; anything else fails at once.
wait_ready() {
	local timeout=${READY_TIMEOUT:-180} start=$SECONDS attempt=0 res status
	[[ $timeout =~ ^[0-9]+$ ]] || { echo "error: READY_TIMEOUT must be a number of seconds" >&2; return 1; }
	while :; do
		attempt=$((attempt + 1))
		res=$(es GET "$1" </dev/null 2>/dev/null) || true
		status=${res##*$'\n'}
		[[ $status =~ ^[0-9]{3}$ ]] || status=000
		case "$status" in
		200) return 0 ;;
		401 | 408 | 503 | 000) ;;
		*) echo "error: GET $1 returned $status: ${res%$'\n'*}" >&2; return 1 ;;
		esac
		if [ $((SECONDS - start)) -ge "$timeout" ]; then
			echo "error: GET $1 not ready after ${timeout}s ($attempt attempts), last answer $status: ${res%$'\n'*}" >&2
			return 1
		fi
		echo "waiting for $1: attempt $attempt answered $status, $((SECONDS - start))s of ${timeout}s" >&2
		sleep 3
	done
}

create_index_once() {
	if [ "$(es GET "/$1" </dev/null | tail -n1)" = 404 ]; then
		es_ok PUT "/$1" >/dev/null
	fi
}

"${compose[@]}" ps --status running -q es01 | grep -q . ||
	{ echo "error: es01 is not running, start it with make up SCENARIO=$(basename "$dir")" >&2; exit 1; }

wait_ready '/_cluster/health?wait_for_status=yellow&timeout=30s'

es_ok PUT /_index_template/app >/dev/null <<'JSON'
{
  "index_patterns": ["app-*"],
  "template": {
    "settings": {"number_of_shards": 1, "number_of_replicas": 0}
  }
}
JSON

es_ok PUT /_security/role/app_writer >/dev/null <<'JSON'
{
  "indices": [
    {"names": ["app-*"], "privileges": ["read", "create_doc"]}
  ]
}
JSON

APP_PASSWORD=$APP_PASSWORD jq -n '{password: env.APP_PASSWORD, roles: ["app_writer"], full_name: "app-x service"}' |
	es_ok PUT /_security/user/app_x >/dev/null

create_index_once app-a </dev/null
create_index_once secret <<'JSON'
{"settings": {"number_of_shards": 1, "number_of_replicas": 0}}
JSON
echo '{"message": "seed", "@timestamp": "2026-01-01T00:00:00Z"}' |
	es_ok PUT '/app-a/_doc/seed?refresh=true' >/dev/null
echo '{"message": "not for app-x", "@timestamp": "2026-01-01T00:00:00Z"}' |
	es_ok PUT '/secret/_doc/1?refresh=true' >/dev/null

# The key is owned by elastic, so role_descriptors is its only limit: without
# them it would be a superuser key. Keys never expire unless told to.
key=$(es_ok POST /_security/api_key <<'JSON' | jq -er .encoded
{
  "name": "app-x-key",
  "expiration": "30d",
  "role_descriptors": {
    "app_x_key": {
      "indices": [
        {"names": ["app-*"], "privileges": ["read", "create_doc"]}
      ]
    }
  }
}
JSON
)

if [ -n "$key_out" ]; then
	(umask 077 && printf 'APP_API_KEY=%s\n' "$key" >"$key_out")
	echo "role app_writer, user app_x and an app API key (30 days, owned by elastic) created; key written to $key_out"
else
	echo "role app_writer, user app_x and an app API key (30 days, owned by elastic) created" >&2
	printf 'APP_API_KEY=%s\n' "$key"
fi

Not tested

Five nodes, and losing nodes one at a time from five.
Two nodes, and two nodes plus a voting-only tiebreaker.
Network partitions, and anything that needs separate hosts.
Growth on OpenSearch, by either procedure, and an OpenSearch three-node cluster.
Kibana with three nodes (elasticsearch.hosts listing all three).
The rollback scripts of both procedures, and the write-blocked window of procedure A.
Time from a node loss to green, measured from the loss.

OpenSearch 3.9.0: what differs

Nothing on this page ran on OpenSearch 3.9.0; the OpenSearch three-node and growth scenarios are not written yet. From the documentation, not from the lab:

Topic	OpenSearch	Basis
Bootstrap setting	`cluster.initial_cluster_manager_nodes`; the role is `cluster_manager`, “master” is deprecated	official, tuning your cluster
Transport TLS when growing	Already in place from day one (see the previous page), so growth has no TLS step. New node certificates will most likely have to come from the same CA and match `plugins.security.nodes_dn`; not checked	not tested
Voting-only tiebreaker	No voting-only role was found in the OpenSearch source on its main branch, and the 3.9 code was not checked; treat the two-plus-tiebreaker topology as Elasticsearch-only until it is	not tested