De Vrije Cloud: Post-Tyranny Tech Multi-Tenant Infrastructure

Infrastructure as Code for our "Vrije Cloud", a scalable multi-tenant VPS platform.

📚 Documentation

→ Read the Complete Documentation (Authentik SSO required)

Quick Links

• New here? Start with the Day One Onboarding Tutorial
• Need to do something? Check the How-To Guides
• Looking up details? Browse the Reference
• Want to understand why? Read the Explanations
• Decisions we've made: Architecture Decision Records

The documentation site is hosted on the onze production server and protected with Authentik SSO, so the links above prompt for login.

Local Documentation Development

cd mkdocs
pip install -r requirements.txt
mkdocs serve  # Opens at http://127.0.0.1:8000

🏗️ Architecture

• Provisioning: OpenTofu (open source Terraform fork)
• Configuration: Ansible with dynamic Hetzner Cloud inventory
• CI/CD: Forgejo Actions (plan on PR, auto-apply for dev/test, manual apply for prod)
• Secrets: SOPS + Age encryption
• Hosting: Hetzner Cloud (EU-based, GDPR-compliant)
• Reverse proxy / TLS: Traefik (per client server, automatic Let's Encrypt)
• Identity: Authentik (OAuth2/OIDC SSO, MIT license)
• Storage: Nextcloud (German company, AGPL 3.0)
• Backups: Restic to Hetzner Storage Box (encrypted, automated daily, per-server sub-accounts)

Distributed by design — no central edge for client traffic

Client traffic is never routed through a shared edge. Each client server runs its own Traefik, its own Authentik (SSO/OIDC), and its own services (Nextcloud, Matrix, Listmonk, …). This gives strong isolation and independence between tenants.

A dedicated edge server exists for platform services only — billing (Paymenter), the Forgejo instance and CI runners, and monitoring (Prometheus/Grafana/Uptime Kuma). It never proxies client traffic.

📁 Repository Structure

infrastructure/
├── environments/            # Per-environment OpenTofu roots (the primary IaC)
│   ├── dev/                 # vrije-cloud-dev   — auto-apply on merge
│   ├── test/                # vrije-cloud-test  — auto-apply on merge
│   ├── prod/                # vrije-cloud-prod  — MANUAL apply only
│   └── edge/                # vrije-cloud-edge  — platform services
├── modules/                 # Reusable OpenTofu modules
│   ├── hetzner-server/      # Server + volume + primary IPs
│   ├── dns-records/         # A / AAAA / wildcard records
│   └── firewall/            # Shared firewall
├── tofu/                    # Shared cloud-init (user-data-public.yml) + legacy monolithic state
├── ansible/                 # Playbooks and roles (30+ roles: traefik, authentik, nextcloud, matrix, …)
│   ├── playbooks/           # setup.yml, deploy.yml, deploy-with-state.yml, …
│   ├── roles/               # One role per service / concern
│   └── hcloud.yml           # Dynamic inventory (discovers servers via the Hetzner API)
├── .forgejo/workflows/      # CI/CD: plan, apply-{dev,test,prod,edge}, server lifecycle, …
├── secrets/                 # SOPS-encrypted secrets (git-safe)
│   ├── shared.sops.yaml     # Shared infrastructure secrets (Hetzner tokens, Mailgun, …)
│   └── clients/             # Per-client secrets + template.sops.yaml
├── keys/                    # Encryption and SSH keys
│   ├── age-key.txt          # SOPS encryption key (gitignored — NEVER commit; store in password manager)
│   └── ssh/                 # SSH keys (private gitignored, public committed)
├── scripts/                 # 40+ management / automation helpers
├── clients/                 # Client registry and documentation
├── mkdocs/                  # Documentation site (Diátaxis; deployed to docs.onze.vrije.cloud)
└── docs/                    # Migration docs, runbooks, and legacy guides

Server counts drift; don't hand-maintain rosters here. Derive the live list of production servers with:

grep '^module "' environments/prod/main.tf | grep -v firewall

🔄 Operational Workflow

Changes flow through Forgejo Actions on the edge Forgejo: forgejo.vrije.cloud/vrije-cloud/infrastructure. A plan runs on every PR; apply behaviour depends on the environment.

Environment	Hetzner project	Apply on merge?
dev	vrije-cloud-dev	✅ automatic
test	vrije-cloud-test	✅ automatic
prod	vrije-cloud-prod	❌ manual trigger only
edge	vrije-cloud-edge	🏗️ platform services

Production Environment

All production changes go through Forgejo Actions with a manual gate:

# 1. Feature branch
git checkout main && git pull
git checkout -b feature/add-production-server

# 2. Edit environments/prod/main.tf (and dns.tf / outputs.tf as needed)

# 3. Commit and push
git add environments/prod/
git commit -m "feat: add new production server"
git push -u origin feature/add-production-server

# 4. Open a PR in Forgejo — the plan workflow runs automatically
#    https://forgejo.vrije.cloud/vrije-cloud/infrastructure

# 5. Merge to main (production does NOT auto-apply)

# 6. Manually trigger the production apply
#    Actions → "Apply to Production" (apply-prod.yaml) → Run workflow
#    Branch: main   ·   Type the confirmation: apply-to-production

# 7. Review the plan in the 120-second window, then let it apply (or cancel)

Production safety features:

• ✅ Manual trigger only — never auto-applies
• ✅ Confirmation required — must type apply-to-production
• ✅ 120-second review window — full plan output with countdown
• ✅ S3 state with versioning — recover from mistakes
• ✅ Cancel anytime — stop before apply completes

Dev and Test Environments

Dev and test changes auto-apply on merge:

git checkout main && git pull
git checkout -b feature/change-dev

# Edit environments/dev/** or environments/test/**
git commit -am "feat: change dev"
git push -u origin feature/change-dev

# Open a PR (plan runs automatically) → merge → apply runs automatically

Path-based routing on merge to main:

• environments/dev/** changed → apply-dev runs
• environments/test/** changed → apply-test runs
• environments/prod/** changed → no auto-apply (manual trigger)
• modules/** changed → dev and test apply (prod still manual)

Automated server lifecycle

A customer order (or an operator dispatch) can drive provisioning end-to-end via server-provision.yaml → server-configure.yaml (dev/test), with server-resize.yaml and server-cancel.yaml for upgrades and teardown. See the how-to guides for details.

No laptop access needed for CI operations — all credentials are stored as Forgejo Actions secrets.

---

🚀 Quick Start

Prerequisites

• OpenTofu >= 1.6
• Ansible >= 2.15
• SOPS + Age
• Hetzner Cloud account
• Hetzner Storage Box — required for backups

Add a New Client Server

The current workflow is per-environment OpenTofu + Forgejo Actions. Full walkthrough: How-To → Add a Client Server.

export SOPS_AGE_KEY_FILE="./keys/age-key.txt"

# 1. Generate SSH keys for the new client
./scripts/generate-client-keys.sh <client>

# 2. Generate ALL application secrets (passwords, tokens, API keys)
#    ⚠️ Do NOT skip — without it the secrets file is only CHANGEME placeholders and deploy fails.
SOPS_AGE_KEY_FILE="keys/age-key.txt" ./scripts/generate-all-secrets.sh <client>

# 3. Add the server to the target environment
#    - environments/<env>/main.tf   (server module)
#    - environments/<env>/dns.tf    (DNS records)
#    - environments/<env>/outputs.tf (monitoring targets)

# 4. Open a PR (plan runs) → merge
#    dev/test auto-apply; prod requires the manual apply-prod trigger.

# 5. Configure the server with Ansible
ansible-playbook -i ansible/hcloud.yml ansible/playbooks/setup.yml  --limit <client>
ansible-playbook -i ansible/hcloud.yml ansible/playbooks/deploy.yml --limit <client>

Secrets are always edited through ./scripts/modify-secrets-safe.sh — never decrypt/edit/encrypt *.sops.yaml by hand. The Hetzner API tokens live encrypted in secrets/shared.sops.yaml and are loaded automatically; there is no need to set HCLOUD_TOKEN manually for the helper scripts.

Decommission a Client Server

DNS records are managed by OpenTofu and cleaned up automatically on destroy.

# 1. Remove the server's module block(s) from environments/<env>/main.tf (and dns.tf / outputs.tf)

# 2. Open a PR — the plan shows the resources to destroy — then merge

# 3. For PRODUCTION servers, first disable delete_protection AND rebuild_protection
#    for each server in the Hetzner Cloud Console (they are on by default in prod)

# 4. Apply:
#    - dev/test: applies automatically on merge
#    - prod:     manually trigger apply-prod.yaml

Important:

• Production servers have delete_protection and rebuild_protection enabled — disable them in the Hetzner Console before the destroy apply, or it will fail.
• OpenTofu removes the DNS records as part of the destroy.

Manual Setup (Advanced)

Click to expand manual, laptop-driven setup

This bypasses Forgejo Actions. Prefer the CI flow above; use this only for local development or recovery.

1. Clone the repository:

   git clone <repo-url>
   cd infrastructure
   

2. Provide the Age key (from the password manager) at keys/age-key.txt:

   export SOPS_AGE_KEY_FILE="./keys/age-key.txt"
   

3. Create client secrets from the template:

   cp secrets/clients/template.sops.yaml secrets/clients/<client>.sops.yaml
   ./scripts/generate-all-secrets.sh <client>
   

4. Provision infrastructure for the target environment:

   cd environments/<env>   # dev | test | prod | edge
   tofu init
   tofu plan               # always review before apply
   tofu apply              # NOTE: prod is normally applied via Forgejo Actions, not from a laptop
   

5. Deploy applications:

   cd ../../ansible
   export HCLOUD_TOKEN="your-token"          # or let the helper scripts load it from SOPS
   export SOPS_AGE_KEY_FILE="../keys/age-key.txt"
   
   ansible-playbook -i hcloud.yml playbooks/setup.yml  --limit <client>
   ansible-playbook -i hcloud.yml playbooks/deploy.yml --limit <client>
   

🎯 Project Principles

1. EU-first: European vendors and data residency
2. Truly open source: Avoid source-available or restrictive licenses
3. Client isolation: Full separation between tenants (no shared edge for client traffic)
4. Infrastructure as Code: All changes via version control
5. Security by default: Encryption, hardening, least privilege

📝 License

MIT License — see LICENSE for details.

🙋 Support

For issues or questions, please create an issue or contact us at vrijecloud@postxsociety.org.