Minimal security, with Caddy in front

Verified against Elasticsearch 9.5.5 and OpenSearch 3.9.0 on 29 October 2026. Configs: github.com/jtsunne/small-elasticsearch-lab at v0.1.0.

For one node, keep authentication on, skip certificates on the node, terminate TLS at Caddy, give applications a scoped, expiring API key, and treat Caddy’s path allowlist as a speed bump, not as access control. I ran every bypass I could think of against this setup. Caddy stopped the path tricks, but a _bulk body can name any index, and an expensive query passes every proxy rule; only the Elasticsearch role stops the first, and nothing on this page stops the second. If Kibana is exposed, its console reaches every API the logged-in user’s role allows, whatever the proxy says.

What I ran: the scenarios es-single (83 of 83 rows), es-single-kibana (90 of 91, see Kibana), os-single (62 of 62) and os-single-dashboards (70 of 70), on 29 October 2026, on my laptop (Linux, cgroup v2, 12 logical CPUs, about 15.5 GiB of RAM, Docker 29.9.1, Compose 5.6.0), with the images pinned in the companion repo’s versions.env. One laptop run is one data point. A row id such as es-single-56 is a line in scenarios/es-single/expected.tsv; its result is in results/tests/2026-10-29-es-single.tap in the companion repo, and results/FINDINGS-2026-10-29.md explains every finding.

What Elasticsearch requires

• Authentication on, always. With xpack.security.enabled: true every request needs credentials; an allowlisted client without them gets 401 (es-single-09, -10).
• Transport TLS only when there is a second node. “Transport SSL must be enabled if security is enabled” is a bootstrap check, and bootstrap checks are not enforced with discovery.type: single-node. [official] The next page adds it when the cluster grows.
• HTTP TLS is optional on a self-managed node. [official] Elastic calls this setup “not sufficient for production mode clusters” and HTTP TLS “highly recommended”. [official] Here Caddy provides HTTPS for everything that leaves the host; the hop from Caddy to the node is plain HTTP on a host-local Docker bridge. That is the trade: no certificates to manage on the node, and anything with access to that bridge can read the traffic.
• Setting any xpack.security.* turns off security auto-configuration. That is intended here: no enrollment tokens, no generated certificates, nothing I did not write down.

The compose file

The node publishes no port. Only Caddy does, on loopback by default. From the host, 127.0.0.1:9200 is refused (es-single-06).

name: es-single

services:
  es01:
    image: ${ES_IMAGE:?ES_IMAGE comes from versions.env}
    environment:
      - node.name=es01
      - cluster.name=es-single
      - discovery.type=single-node
      - xpack.security.enabled=true
      # Allowed without node certificates only because discovery.type is
      # single-node; a second node needs transport TLS.
      - xpack.security.http.ssl.enabled=false
      - xpack.security.transport.ssl.enabled=false
      - xpack.license.self_generated.type=basic
      - bootstrap.memory_lock=true
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD:?set ELASTIC_PASSWORD in .env}
      # Same value auto-sizing picks for a 2 GiB container, set explicitly so the
      # OpenSearch scenarios (which do not auto-size) run the same heap. Set
      # ES_HEAP_OPTS to an empty string to let Elasticsearch size it (L-M1).
      - ES_JAVA_OPTS=${ES_HEAP_OPTS--Xms1g -Xmx1g}
    cpus: 1
    mem_limit: 2g
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65535
        hard: 65535
    volumes:
      - esdata:/usr/share/elasticsearch/data
    networks:
      - backend
    healthcheck:
      test: ["CMD-SHELL", "curl -s http://localhost:9200 | grep -q 'missing authentication credentials'"]
      interval: 10s
      timeout: 10s
      retries: 30
      start_period: 30s

  caddy:
    image: ${CADDY_IMAGE:?CADDY_IMAGE comes from versions.env}
    depends_on:
      es01:
        condition: service_healthy
    environment:
      - ES_SITE_ADDRESS=${ES_SITE_ADDRESS:?set ES_SITE_ADDRESS in .env}
      - ALLOWED_CIDRS=${ALLOWED_CIDRS:?set ALLOWED_CIDRS in .env}
    ports:
      - "${CADDY_BIND:-127.0.0.1}:${CADDY_HTTP_PORT:-80}:80"
      - "${CADDY_BIND:-127.0.0.1}:${CADDY_HTTPS_PORT:-443}:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    networks:
      - backend

networks:
  backend: {}

volumes:
  esdata: {}
  caddy_data: {}
  caddy_config: {}

Tested at v0.1.0: scenarios/es-single/compose.yaml.

Three choices in it are worth a sentence each:

• ES_JAVA_OPTS=-Xms1g -Xmx1g is what auto-sizing picks for a 2 GiB container anyway. I set it so the OpenSearch scenario, which does not size its heap from the container, runs the same heap. Without it, Elasticsearch 9.5.5 gave the node exactly 1 GiB by itself (es-single-02, auto-sizing run). Elastic does not recommend ES_JAVA_OPTS in production; there, keep the default sizing.
• No node.processors: with cpus: 1 the node reported 1 allocated processor, a write pool of 1 and a search pool of 2 (es-single-03, -04), so the quota is honoured on cgroup v2.
• The healthcheck greps for missing authentication credentials, as Elastic’s own compose file does. It proves the node answers HTTP, not that it is ready; see Readiness.

vm.max_map_count is a host setting. Elasticsearch asks for 1048576 [official]; with discovery.type: single-node the check is not enforced, but set it anyway, because the next page needs it.

The Caddyfile

# Requires Caddy 2.11.1 or later (CVE-2026-27587). The {$VAR:default} values let
# the same file run in compose, in CI validation and in the proxy-only test
# harness; compose always sets ES_SITE_ADDRESS and ALLOWED_CIDRS.
{
	admin {$CADDY_ADMIN:localhost:2019}
	servers {
		protocols h1 h2
	}
}

{$ES_SITE_ADDRESS:es.localhost} {
	log

	# route keeps the written order. Without it Caddy sorts directives and runs
	# handle and reverse_proxy before abort, so the IP allowlist would not
	# protect the proxied paths.
	route {
		# remote_ip is the immediate TCP peer. Behind another proxy or Docker's
		# userland proxy that is not the client, and this check means nothing.
		@outside not remote_ip {$ALLOWED_CIDRS:127.0.0.1/32}
		abort @outside

		request_body {
			max_size 20MB
		}

		# Matched against the raw request URI, because reverse_proxy forwards the
		# raw path: path and path_regexp see a decoded, dot-cleaned path, so
		# /secret/../app-a/_search would match them and still reach the node as
		# written. Any encoding, dot segment, double slash, upper case, comma,
		# wildcard, date math or remote-cluster prefix fails the pattern and
		# gets the 404 below.
		# This allowlist is a speed bump: a _bulk body can still name any index,
		# and only the Elasticsearch role decides what the caller may touch.
		@app_read {
			method GET POST
			expression `{http.request.uri}.matches("^/app-[a-z0-9][a-z0-9-]*/_(search|count)([?].*)?$")`
		}
		@app_write {
			method POST PUT
			expression `{http.request.uri}.matches("^/app-[a-z0-9][a-z0-9-]*/_bulk([?].*)?$")`
		}

		reverse_proxy @app_read {$ES_UPSTREAM:es01:9200} {
			# Elasticsearch does not honour this header; dropping it only keeps
			# the upstream request equal to what the method matcher saw.
			header_up -X-HTTP-Method-Override
		}
		reverse_proxy @app_write {$ES_UPSTREAM:es01:9200} {
			header_up -X-HTTP-Method-Override
		}

		respond 404
	}
}

Tested at v0.1.0: scenarios/es-single/Caddyfile.

Caddy 2.11.1 or later is required because of CVE-2026-27587, a path matcher bypass with %xx in patterns; the lab pins 2.11.7. [official]

Two things in this file are there because the obvious version is broken.

Order. Caddy does not run directives in the order you write them. It sorts them, and handle and reverse_proxy come before abort. A top-level @denied not remote_ip ... with abort @denied, followed by handle blocks that proxy, lets every client on the internet reach the proxied paths. [official] The companion repo’s proxy-only harness (Caddy 2.11.7 built locally, an echo server as upstream) showed it: with that structure a non-allowlisted client reached /app-a/_search and /app-a/_bulk. Everything inside one route block keeps the written order, so the allowlist runs first.

Raw path versus cleaned path. The path and path_regexp matchers see a decoded, dot-cleaned path. reverse_proxy forwards the request-target exactly as the client sent it. So a matcher can approve one path while the node receives another:

The “forwarded” column comes from the proxy-only harness, not from a run against Elasticsearch: eight such rows reached the echo upstream unchanged with a path_regexp version of this file (tests/README.md, fixture Caddyfile.plan). Whether Elasticsearch would then route any of them somewhere harmful was not tested. The fix is to match what is forwarded: an expression matcher on {http.request.uri}, the raw path plus query string. Any encoding, dot segment, double slash, upper case, comma, wildcard or date math then fails the pattern. Clients that send canonical paths do not notice.

What the proxy stopped, and what it did not

Every row below ran through Caddy against Elasticsearch 9.5.5, with the app’s API key unless noted. “Node” is the same request sent straight to es01:9200 on the compose network, to show which layer would stop it if Caddy were not there.

Three rows are the reason this page exists:

• The bulk body names its own index. Caddy sees /app-a/_bulk and forwards it. The body says secret. Only the role answers 403, per item, inside a 200 response. A client that only checks the HTTP status sees success.
• Expensive queries pass. Aggregations, scripts and runtime mappings are body content. No proxy rule here looks at them, and I did not test any Elasticsearch-side limit on them.
• The body limit is not a wall. In the proxy-only harness, Caddy answered 413 for a body over 20 MB but had already streamed the first 20,000,000 bytes to the upstream, which then saw the request end early. Whether Elasticsearch discards such a truncated request was not tested.

What a blocked client sees

abort sends no HTTP response at all. What the client reports depends on the protocol. Over HTTP/2, which every HTTPS client here negotiated, Caddy resets the stream and curl exits with 92 (HTTP/2 stream ... was not closed cleanly). Over HTTP/1.1, Caddy closes the connection and curl exits with 52 (empty reply), or 55 or 56 when the reset arrives while sending or reading. My own test runner first counted only 52 to 56 as “aborted” and reported the HTTP/2 aborts as failures (es-single-07, -08, -65 in the first run that day). If you monitor for blocked clients, look for resets, not for a status code.

Users, roles and API keys

The application gets a role with read and create_doc on app-* and no cluster privileges, and a user app_x with that role. Its API key is created by elastic at setup, with an expiration and role_descriptors; scenarios/es-single/setup-roles.sh sends this request:

POST /_security/api_key
{
  "name": "app-x-key",
  "expiration": "30d",
  "role_descriptors": {
    "app_x_key": {
      "indices": [
        {"names": ["app-*"], "privileges": ["read", "create_doc"]}
      ]
    }
  }
}

Why it looks like this, from the create API key page: “By default, API keys never expire”, and with role_descriptors “the resultant permissions are an intersection of API keys permissions and the authenticated user’s permissions”. [official] The creator is elastic, so role_descriptors are the key’s only limit. Send the same request without them and you have made a superuser key.

What the lab showed for the key:

• It authenticates as api_key named app-x-key, owned by elastic (es-single-81).
• The stored key has an expiration and exactly the app_x_key descriptor (es-single-82).
• Neither app_x nor the key can create another key: 403 security_exception (es-single-79, -80). A leaked app password cannot mint keys.

The client sends Authorization: ApiKey <encoded>, where encoded is the field of the same name in the response. Rotation: create a new key, switch the application, then invalidate the old one by id as elastic.

create_doc does not allow the bulk index action

This one surprised me. create_doc is meant for “append only”: new documents, never overwrites. I expected it to cover any write without an _id. On Elasticsearch 9.5.5 it does not:

The reason for the bulk index item:

action [indices:data/write/index:op_type/index] is unauthorized ... this action is granted by the index privileges [create,index,write,all]

Elasticsearch authorizes a bulk item by its operation type, not by whether the id is generated. If your client’s bulk helper sends index by default, configure it to send create. Granting index or write instead makes the error go away and also lets the key overwrite any document by id, which is what create_doc exists to prevent. create_doc does not create indices either (es-single-60).

Kibana

es-single-kibana adds Kibana 9.5.5 on a second Caddy site with the same IP allowlist and no path rules, because Kibana needs its whole URL space. Kibana connects to http://es01:9200 directly as kibana_system, not through Caddy. Kibana’s own login does the authentication.

# Requires Caddy 2.11.1 or later (CVE-2026-27587). The {$VAR:default} values let
# the same file run in compose, in CI validation and in the proxy-only test
# harness; compose always sets the site addresses and ALLOWED_CIDRS.
{
	admin {$CADDY_ADMIN:localhost:2019}
	servers {
		protocols h1 h2
	}
}

{$ES_SITE_ADDRESS:es.localhost} {
	log

	# route keeps the written order. Without it Caddy sorts directives and runs
	# handle and reverse_proxy before abort, so the IP allowlist would not
	# protect the proxied paths.
	route {
		# remote_ip is the immediate TCP peer. Behind another proxy or Docker's
		# userland proxy that is not the client, and this check means nothing.
		@outside not remote_ip {$ALLOWED_CIDRS:127.0.0.1/32}
		abort @outside

		request_body {
			max_size 20MB
		}

		# Matched against the raw request URI, because reverse_proxy forwards the
		# raw path: path and path_regexp see a decoded, dot-cleaned path, so
		# /secret/../app-a/_search would match them and still reach the node as
		# written. Any encoding, dot segment, double slash, upper case, comma,
		# wildcard, date math or remote-cluster prefix fails the pattern and
		# gets the 404 below.
		# This allowlist is a speed bump: a _bulk body can still name any index,
		# and only the Elasticsearch role decides what the caller may touch.
		@app_read {
			method GET POST
			expression `{http.request.uri}.matches("^/app-[a-z0-9][a-z0-9-]*/_(search|count)([?].*)?$")`
		}
		@app_write {
			method POST PUT
			expression `{http.request.uri}.matches("^/app-[a-z0-9][a-z0-9-]*/_bulk([?].*)?$")`
		}

		reverse_proxy @app_read {$ES_UPSTREAM:es01:9200} {
			# Elasticsearch does not honour this header; dropping it only keeps
			# the upstream request equal to what the method matcher saw.
			header_up -X-HTTP-Method-Override
		}
		reverse_proxy @app_write {$ES_UPSTREAM:es01:9200} {
			header_up -X-HTTP-Method-Override
		}

		respond 404
	}
}

# Kibana is a full Elasticsearch client for anyone who can log in (Dev Tools
# console), so the path allowlist above does not apply to Kibana users; their
# Elasticsearch roles do. Caddy only adds the IP allowlist and TLS here.
{$KIBANA_SITE_ADDRESS:kibana.localhost} {
	log

	route {
		@outside not remote_ip {$ALLOWED_CIDRS:127.0.0.1/32}
		abort @outside

		reverse_proxy {$KIBANA_UPSTREAM:kibana:5601}
	}
}

Tested at v0.1.0: scenarios/es-single-kibana/Caddyfile.

Kibana’s console goes around the allowlist. Kibana sends Dev Tools console requests to Elasticsearch directly, with the logged-in user’s credentials. Through POST /api/console/proxy?path=_cluster/settings&method=GET, elastic got 200 with the cluster settings (es-single-kibana-83), although Caddy answers 404 for /_cluster/settings on the Elasticsearch site (es-single-kibana-14). app_x was refused with 401 or 403; the run did not record which (es-single-kibana-84). Whoever can reach the Kibana site and log in gets every Elasticsearch API their role allows. Give Kibana users roles as narrow as the API key’s, keep elastic for setup, and keep the Kibana site’s allowlist as tight as the Elasticsearch site’s.

The console proxy is an internal route in Kibana 9.5.5. Without the header x-elastic-internal-origin it answers:

400 uri [/api/console/proxy] with method [post] exists but is not available with the current configuration

With the header, any value, the request goes through (es-single-kibana-91 and -83). Kibana’s own browser client sends it on every request. It is not a security boundary.

Other Kibana notes from the run:

• /api/status answers 200 without credentials once Kibana is ready, with available in the body; /login answers 200 for an allowlisted client; both are aborted for anyone else (es-single-kibana-79 to -82).
• Right after start, while Kibana created its system indices, the node was yellow for a moment: one primary unassigned (es-single-kibana-01, the one failed row of the run). Wait for green with a timeout instead of checking once.
• The container ran with 1024 MB, as in Elastic’s compose file; Elastic recommends at least 2048 MB with alerting. [official] I did not measure Kibana’s memory use or test alerting.
• Set xpack.encryptedSavedObjects.encryptionKey, and keep it stable: without it Kibana generates one per start and some features, alerting among them, are unavailable.

Readiness is not “healthy”

docker compose up --wait returns when the healthcheck passes, and the healthcheck only proves that the node answers HTTP. On the three-node scenario the nodes were “healthy” while the .security-7 primary was still unassigned, and the setup script’s first call as elastic failed:

503 unavailable_shards_exception: at least one primary shard for the index [.security-7] is unavailable
authentication_processing_error: failed to retrieve password hash for reserved user [elastic]

My first run carried on without an app key and 25 rows failed. The setup scripts now wait for cluster health and retry 503, 401, 408 and no answer for up to 180 seconds, printing one line per attempt. Any script that runs right after up needs the same loop, on one node too.

Generate the files

The config generator writes the files of this page’s scenarios: one node, with or without Kibana or Dashboards. For the inputs the lab ran, its output is the tested files, apart from a short generated header. Changed values such as memory or host names keep the tested structure and are listed in the file header; a structural change, such as a disabled route or an IPv6 allowlist, marks the output as not tested. Without JavaScript it shows the files of es-single, the scenario this page walks through.

# Copy to .env, or run ./init-env.sh, which writes .env with random passwords.
# Passwords: alphanumeric, at least 6 characters (Elasticsearch minimum); the
# scripts here refuse the change-me placeholders.
ELASTIC_PASSWORD=change-me-elastic
APP_PASSWORD=change-me-app

# Lab defaults: Caddy on loopback, high ports, Caddy's internal CA for *.localhost.
# On a real host: CADDY_BIND=0.0.0.0, ports 80 and 443, ES_SITE_ADDRESS set to a
# public DNS name so Caddy obtains a certificate.
CADDY_BIND=127.0.0.1
CADDY_HTTP_PORT=8080
CADDY_HTTPS_PORT=8443
ES_SITE_ADDRESS=es.localhost

# Space-separated CIDRs. Caddy sees the immediate TCP peer, see README.md.
ALLOWED_CIDRS=127.0.0.1/32

# Unset: -Xms1g -Xmx1g. Empty string: Elasticsearch sizes the heap itself.
#ES_HEAP_OPTS=

# Requires Caddy 2.11.1 or later (CVE-2026-27587). The {$VAR:default} values let
# the same file run in compose, in CI validation and in the proxy-only test
# harness; compose always sets ES_SITE_ADDRESS and ALLOWED_CIDRS.
{
	admin {$CADDY_ADMIN:localhost:2019}
	servers {
		protocols h1 h2
	}
}

{$ES_SITE_ADDRESS:es.localhost} {
	log

	# route keeps the written order. Without it Caddy sorts directives and runs
	# handle and reverse_proxy before abort, so the IP allowlist would not
	# protect the proxied paths.
	route {
		# remote_ip is the immediate TCP peer. Behind another proxy or Docker's
		# userland proxy that is not the client, and this check means nothing.
		@outside not remote_ip {$ALLOWED_CIDRS:127.0.0.1/32}
		abort @outside

		request_body {
			max_size 20MB
		}

		# Matched against the raw request URI, because reverse_proxy forwards the
		# raw path: path and path_regexp see a decoded, dot-cleaned path, so
		# /secret/../app-a/_search would match them and still reach the node as
		# written. Any encoding, dot segment, double slash, upper case, comma,
		# wildcard, date math or remote-cluster prefix fails the pattern and
		# gets the 404 below.
		# This allowlist is a speed bump: a _bulk body can still name any index,
		# and only the Elasticsearch role decides what the caller may touch.
		@app_read {
			method GET POST
			expression `{http.request.uri}.matches("^/app-[a-z0-9][a-z0-9-]*/_(search|count)([?].*)?$")`
		}
		@app_write {
			method POST PUT
			expression `{http.request.uri}.matches("^/app-[a-z0-9][a-z0-9-]*/_bulk([?].*)?$")`
		}

		reverse_proxy @app_read {$ES_UPSTREAM:es01:9200} {
			# Elasticsearch does not honour this header; dropping it only keeps
			# the upstream request equal to what the method matcher saw.
			header_up -X-HTTP-Method-Override
		}
		reverse_proxy @app_write {$ES_UPSTREAM:es01:9200} {
			header_up -X-HTTP-Method-Override
		}

		respond 404
	}
}

name: es-single

services:
  es01:
    image: ${ES_IMAGE:?ES_IMAGE comes from versions.env}
    environment:
      - node.name=es01
      - cluster.name=es-single
      - discovery.type=single-node
      - xpack.security.enabled=true
      # Allowed without node certificates only because discovery.type is
      # single-node; a second node needs transport TLS.
      - xpack.security.http.ssl.enabled=false
      - xpack.security.transport.ssl.enabled=false
      - xpack.license.self_generated.type=basic
      - bootstrap.memory_lock=true
      - ELASTIC_PASSWORD=${ELASTIC_PASSWORD:?set ELASTIC_PASSWORD in .env}
      # Same value auto-sizing picks for a 2 GiB container, set explicitly so the
      # OpenSearch scenarios (which do not auto-size) run the same heap. Set
      # ES_HEAP_OPTS to an empty string to let Elasticsearch size it (L-M1).
      - ES_JAVA_OPTS=${ES_HEAP_OPTS--Xms1g -Xmx1g}
    cpus: 1
    mem_limit: 2g
    ulimits:
      memlock:
        soft: -1
        hard: -1
      nofile:
        soft: 65535
        hard: 65535
    volumes:
      - esdata:/usr/share/elasticsearch/data
    networks:
      - backend
    healthcheck:
      test: ["CMD-SHELL", "curl -s http://localhost:9200 | grep -q 'missing authentication credentials'"]
      interval: 10s
      timeout: 10s
      retries: 30
      start_period: 30s

  caddy:
    image: ${CADDY_IMAGE:?CADDY_IMAGE comes from versions.env}
    depends_on:
      es01:
        condition: service_healthy
    environment:
      - ES_SITE_ADDRESS=${ES_SITE_ADDRESS:?set ES_SITE_ADDRESS in .env}
      - ALLOWED_CIDRS=${ALLOWED_CIDRS:?set ALLOWED_CIDRS in .env}
    ports:
      - "${CADDY_BIND:-127.0.0.1}:${CADDY_HTTP_PORT:-80}:80"
      - "${CADDY_BIND:-127.0.0.1}:${CADDY_HTTPS_PORT:-443}:443"
    volumes:
      - ./Caddyfile:/etc/caddy/Caddyfile:ro
      - caddy_data:/data
      - caddy_config:/config
    networks:
      - backend

networks:
  backend: {}

volumes:
  esdata: {}
  caddy_data: {}
  caddy_config: {}

#!/usr/bin/env bash
# Writes .env next to this script from .env.example with random alphanumeric
# passwords. Refuses to overwrite an existing .env.
set -euo pipefail

dir=$(cd "$(dirname "$0")" && pwd)
env_file="$dir/.env"

if [ -e "$env_file" ]; then
	echo "error: $env_file exists; remove it first to regenerate" >&2
	exit 1
fi

random_secret() {
	LC_ALL=C tr -dc 'A-Za-z0-9' </dev/urandom | head -c "${1:-32}" || true
}

umask 077
while IFS= read -r line; do
	case "$line" in
	*=change-me*) printf '%s=%s\n' "${line%%=*}" "$(random_secret 32)" ;;
	*) printf '%s\n' "$line" ;;
	esac
done <"$dir/.env.example" >"$env_file"
echo "wrote $env_file (mode 600)"

#!/usr/bin/env bash
# Creates the index template, the test indices, the app_writer role and the
# app_x user, then creates an expiring, scoped API key for the application as
# elastic. Run it again to rotate: it creates a new key, see README.md.
#
# Usage: setup-roles.sh [--key-out FILE]
# READY_TIMEOUT (seconds, default 180) bounds the wait for the cluster.
#
# The node publishes no port and Caddy does not proxy _security, so every call
# runs curl inside the es01 container against 127.0.0.1:9200. Passwords reach
# that curl on stdin and as a curl config read from a pipe, never through argv,
# which any user on the host can read with ps.
set -euo pipefail

dir=$(cd "$(dirname "$0")" && pwd)
repo=$(cd "$dir/../.." && pwd)
key_out=""

while [ $# -gt 0 ]; do
	case "$1" in
	--key-out) key_out="${2:?--key-out needs a file}"; shift 2 ;;
	-h | --help) sed -n '2,6p' "$0"; exit 0 ;;
	*) echo "error: unknown argument $1" >&2; exit 2 ;;
	esac
done

command -v jq >/dev/null || { echo "error: jq is required" >&2; exit 1; }
[ -f "$dir/.env" ] || { echo "error: $dir/.env missing, run init-env.sh" >&2; exit 1; }

# Read single keys instead of sourcing .env: ALLOWED_CIDRS holds spaces and
# compose does not quote values.
env_value() {
	local v
	v=$(grep -m1 "^$1=" "$dir/.env" | cut -d= -f2-) || true
	printf '%s' "${!1:-$v}"
}
ELASTIC_PASSWORD=$(env_value ELASTIC_PASSWORD)
APP_PASSWORD=$(env_value APP_PASSWORD)
for name in ELASTIC_PASSWORD APP_PASSWORD; do
	case "${!name}" in
	"" | change-me*) echo "error: $name is unset or still the .env.example placeholder" >&2; exit 1 ;;
	esac
done

compose=(docker compose --project-directory "$dir" --env-file "$repo/versions.env"
	--env-file "$dir/.env" -f "$dir/compose.yaml")

# The first stdin line is the password, the rest is the request body. bash
# reads a pipe byte by byte, so curl gets the stream right after that line.
# shellcheck disable=SC2016 # expanded inside the container
remote='IFS= read -r pass; pass=${pass//\\/\\\\}; pass=${pass//\"/\\\"}
curl -sS -w "\n%{http_code}" -X "$1" -H "Content-Type: application/json" \
	--data-binary @- -K <(printf "user = \"%s:%s\"\n" "$3" "$pass") \
	"http://127.0.0.1:9200$2"'

# es METHOD PATH < body; runs as elastic, prints the response body, then the
# status on its own line.
es() {
	{ printf '%s\n' "$ELASTIC_PASSWORD"; cat; } |
		"${compose[@]}" exec -T es01 bash -c "$remote" _ "$1" "$2" elastic
}

# es_ok METHOD PATH < body; prints the body, fails unless 2xx.
es_ok() {
	local res status body
	res=$(es "$@")
	status=${res##*$'\n'}
	body=${res%$'\n'*}
	case "$status" in
	2??) printf '%s' "$body" ;;
	*) echo "error: $1 $2 returned $status: $body" >&2; return 1 ;;
	esac
}

# wait_ready PATH: GET PATH until it answers 200. compose up --wait returns
# when the node answers HTTP, which can be before the .security-7 primary is
# assigned; until then even elastic fails with 503 ("failed to retrieve
# password hash for reserved user"), or 401. 401, 408 (health timeout), 503
# and no answer are retried until READY_TIMEOUT seconds (default 180) have
# passed; anything else fails at once.
wait_ready() {
	local timeout=${READY_TIMEOUT:-180} start=$SECONDS attempt=0 res status
	[[ $timeout =~ ^[0-9]+$ ]] || { echo "error: READY_TIMEOUT must be a number of seconds" >&2; return 1; }
	while :; do
		attempt=$((attempt + 1))
		res=$(es GET "$1" </dev/null 2>/dev/null) || true
		status=${res##*$'\n'}
		[[ $status =~ ^[0-9]{3}$ ]] || status=000
		case "$status" in
		200) return 0 ;;
		401 | 408 | 503 | 000) ;;
		*) echo "error: GET $1 returned $status: ${res%$'\n'*}" >&2; return 1 ;;
		esac
		if [ $((SECONDS - start)) -ge "$timeout" ]; then
			echo "error: GET $1 not ready after ${timeout}s ($attempt attempts), last answer $status: ${res%$'\n'*}" >&2
			return 1
		fi
		echo "waiting for $1: attempt $attempt answered $status, $((SECONDS - start))s of ${timeout}s" >&2
		sleep 3
	done
}

create_index_once() {
	if [ "$(es GET "/$1" </dev/null | tail -n1)" = 404 ]; then
		es_ok PUT "/$1" >/dev/null
	fi
}

"${compose[@]}" ps --status running -q es01 | grep -q . ||
	{ echo "error: es01 is not running, start it with make up SCENARIO=$(basename "$dir")" >&2; exit 1; }

wait_ready '/_cluster/health?wait_for_status=yellow&timeout=30s'

es_ok PUT /_index_template/app >/dev/null <<'JSON'
{
  "index_patterns": ["app-*"],
  "template": {
    "settings": {"number_of_shards": 1, "number_of_replicas": 0}
  }
}
JSON

es_ok PUT /_security/role/app_writer >/dev/null <<'JSON'
{
  "indices": [
    {"names": ["app-*"], "privileges": ["read", "create_doc"]}
  ]
}
JSON

APP_PASSWORD=$APP_PASSWORD jq -n '{password: env.APP_PASSWORD, roles: ["app_writer"], full_name: "app-x service"}' |
	es_ok PUT /_security/user/app_x >/dev/null

create_index_once app-a </dev/null
create_index_once secret <<'JSON'
{"settings": {"number_of_shards": 1, "number_of_replicas": 0}}
JSON
echo '{"message": "seed", "@timestamp": "2026-01-01T00:00:00Z"}' |
	es_ok PUT '/app-a/_doc/seed?refresh=true' >/dev/null
echo '{"message": "not for app-x", "@timestamp": "2026-01-01T00:00:00Z"}' |
	es_ok PUT '/secret/_doc/1?refresh=true' >/dev/null

# The key is owned by elastic, so role_descriptors is its only limit: without
# them it would be a superuser key. Keys never expire unless told to.
key=$(es_ok POST /_security/api_key <<'JSON' | jq -er .encoded
{
  "name": "app-x-key",
  "expiration": "30d",
  "role_descriptors": {
    "app_x_key": {
      "indices": [
        {"names": ["app-*"], "privileges": ["read", "create_doc"]}
      ]
    }
  }
}
JSON
)

if [ -n "$key_out" ]; then
	(umask 077 && printf 'APP_API_KEY=%s\n' "$key" >"$key_out")
	echo "role app_writer, user app_x and an app API key (30 days, owned by elastic) created; key written to $key_out"
else
	echo "role app_writer, user app_x and an app API key (30 days, owned by elastic) created" >&2
	printf 'APP_API_KEY=%s\n' "$key"
fi

Threat model

Covers: the node is reachable only through Caddy; only listed client addresses reach Caddy’s routes; requests without valid credentials get 401; the app key can read app-* and add documents there and do nothing else: no overwrite, no delete, no other index, no cluster settings, no security APIs; keys expire.

Does not cover: a leaked key or password within its own scope; expensive queries; anything running on the host; Kibana users (their role is their limit); traffic between Caddy and the node, which is plaintext on the Docker bridge.

Anti-patterns

• xpack.security.enabled: false because “the proxy handles it”. The bulk row above is why not.
• Publishing 9200 “just for debugging”. As far as I know, Docker writes its own firewall rules for published ports, so a host firewall such as ufw may not apply to them; I did not test that, only that an unpublished port is refused.
• Applications using elastic, or an API key created without role_descriptors.
• A top-level abort next to handle blocks, or path_regexp matchers in front of reverse_proxy.
• Adding Docker’s bridge range to the allowlist. remote_ip is the immediate TCP peer; with Docker’s userland proxy, connections from the host’s own loopback may arrive from the bridge gateway, which would let every local process in. That is a hypothesis; not tested, so check what Caddy logs as remote_ip before you change the list.

Not tested

• The Caddy IP allowlist with Docker’s userland proxy, and the ufw variant of a published port.
• Kibana refusing elastic as its connection user: read from Kibana’s source, not run.
• Which status, 401 or 403, app_x gets from Kibana’s console proxy.
• Whether Elasticsearch discards a request body that Caddy cut off at the size limit.
• Any limit on expensive queries inside Elasticsearch.