Complete homelab post-sprint: SSO docs, monitoring scripts, phase 0/1 closure.

Consolidate sprint status into handoff docs, add Listmonk/Mattermost/Mailcow and Vikunja SSO guides, Beszel alerts script, mattermost inventory, and mark phases 0–1 complete with phase 2 backlog for edge Caddy and security. Co-authored-by: Cursor <cursoragent@cursor.com>
2026-05-24 12:09:56 -04:00 · 2026-05-24 12:09:56 -04:00 · 0f34c51fc8
commit 0f34c51fc8
parent b3f2d438e5
18 changed files with 572 additions and 161 deletions
--- a/4
+++ b/4
@ -711,6 +711,10 @@ beszel-setup-smtp: ## Configure Beszel SMTP (needs BESZEL_EMAIL, BESZEL_PASSWORD
 	@chmod +x scripts/beszel-setup-smtp.sh
 	@./scripts/beszel-setup-smtp.sh

+beszel-setup-alerts: ## Enable Beszel alerts on all systems (needs BESZEL_EMAIL, BESZEL_PASSWORD)
+	@chmod +x scripts/beszel-setup-alerts.sh
+	@./scripts/beszel-setup-alerts.sh
+
 caddy-monitoring: require-ansible ## stats.levkin.ca + status.levkin.ca on Caddy VM
 	@echo "Ensuring monitoring public proxies on caddy..."
 	$(ANSIBLE_PLAYBOOK) playbooks/caddy-monitoring-sites.yml $(ANSIBLE_ARGS)
--- a/docs/guides/cursor-mcp-homelab.md
+++ b/docs/guides/cursor-mcp-homelab.md
@ -0,0 +1,81 @@
+# Cursor MCP servers for this homelab / Ansible repo
+
+**Global config:** `~/.cursor/mcp.json` (all projects)  
+**Project config:** `.cursor/mcp.json` (this repo only — optional override)
+
+After editing, **restart Cursor** or use **Settings → MCP → Refresh**.
+
+---
+
+## Installed (global)
+
+| MCP | Purpose | When the agent uses it |
+|-----|---------|-------------------------|
+| **playwright** | Browser automation (login flows, SSO smoke tests, UI screenshots) | Verify `auth.levkin.ca` → app OIDC; Kuma/Beszel/Listmonk admin clicks |
+| **hermes** | Telegram/Discord/Slack/WhatsApp/Signal/Matrix via Hermes VM **117** | Notify you on deploy finish, alert failures, ask approval from phone |
+
+### Playwright notes
+
+- Official package: `@playwright/mcp@latest` via `npx`
+- Homelab origins restricted in `args` (`*.levkin.ca`, `10.0.10.*`)
+- For saved login state: add `--storage-state=~/path/auth-state.json` after manual login once
+- Headless (no window): add `--headless` to `args`
+
+**If MCP shows “errored”:** Settings → MCP → playwright → view log; restart Cursor.
+
+**Browsers for `@playwright/mcp`:** the MCP package downloads Chromium on first run. You do **not** need `npm install @playwright/test` in this Ansible repo.
+
+Optional local install (only if you run Playwright scripts in-repo):
+
+```bash
+npm install -D @playwright/test
+npx playwright install chromium
+```
+
+### Hermes notes
+
+- Runs over SSH to `ladmin@10.0.10.36` — requires VPN/LAN or Tailscale to Hermes VM
+- Complements Ansible (infra) with **human notifications**, not provisioning
+
+---
+
+## Recommended additions (not installed yet)
+
+| MCP | Why for Ansible / homelab | Install hint |
+|-----|---------------------------|--------------|
+| **GitHub** (`gh` / official) | PRs, CI failures, issue links from chat | Cursor MCP directory → GitHub |
+| **Gitea** (custom or HTTP) | Your `git.levkin.ca` — same as GitHub MCP pattern | Community server or REST via script |
+| **Filesystem** (built-in) | Already available in agent mode | — |
+| **Postgres** | Query listmonk/cal DBs for debugging | `@modelcontextprotocol/server-postgres` + DSN in env |
+| **Docker** | Inspect containers on monitoring/identity LXCs | SSH + `docker` often enough; MCP optional |
+| **Grafana/Prometheus** | If you add observability later | Official or community MCP |
+| **UniFi** | DHCP/client status without opening UI | Community UniFi MCP + `UNIFI_API_KEY` in env |
+| **Proxmox** | VM/LXC state from chat | Community proxmox MCP or keep using `make` + SSH |
+
+**Lower priority:** Notion, Linear, Sentry — only if you adopt those tools.
+
+---
+
+## What MCP does *not* replace
+
+| Task | Use instead |
+|------|-------------|
+| Provision LXCs/VMs | Ansible playbooks + `make` |
+| Secrets | Ansible Vault (`make edit-group-vault`) |
+| Authentik providers/apps | Authentik API token or blueprints (`roles/cal_sso`) |
+| Repeatable SSO | API/blueprints > Playwright (Playwright = verify UI) |
+
+---
+
+## Security
+
+- Do **not** put vault passwords or API tokens in `mcp.json` unless the server supports env vars and you use OS keychain
+- Rotate tokens if pasted in chat
+- Hermes SSH key: same trust as any admin SSH to homelab
+
+---
+
+## Related
+
+- [handoff-2026-05-24.md](handoff-2026-05-24.md)
+- [sso-selfhosted-matrix.md](sso-selfhosted-matrix.md)
--- a/docs/guides/handoff-2026-05-24.md
+++ b/docs/guides/handoff-2026-05-24.md
@ -1,34 +1,53 @@
-# Handoff — homelab sprint (2026-05-24)
+# Homelab sprint handoff — 2026-05-24 (archive)

-**Branch:** `homelab-sprint-2026-05-23` (or current working branch)  
-**Repo:** `/Users/macilia/Documents/code/ansible`  
-**Master plan:** [levkin-selfhost-plan-2.md](levkin-selfhost-plan-2.md) · **Status:** [homelab-status-2026-05-22.md](homelab-status-2026-05-22.md)
+**Branch:** `homelab/post-sprint-2026-05-24` (merge to `master`)  
+**Next work:** [handoff-next-steps.md](handoff-next-steps.md)  
+**Master plan:** [levkin-selfhost-plan-2.md](levkin-selfhost-plan-2.md) · **MCPs:** [cursor-mcp-homelab.md](cursor-mcp-homelab.md)
+
+## Phases 0 + 1 — complete ✅
+
+| Phase | Done |
+|-------|------|
+| **0** | LXCs static, UniFi VM DHCP, DNS apex + auth, identity LXC provisioned |
+| **1** | Authentik @ `.21`, `auth.levkin.ca`, admin + TOTP, Caddy passthrough |

 ---

-## Done this session
+## Done (P0 sprint)

 ### Monitoring (LXC 218 @ `10.0.10.22`)

 | Item | State |
 |------|--------|
 | **Kuma** | 17 HTTP monitors, all email-linked; admin password in vault |
-| **Kuma status page** | https://status.levkin.ca — slug `homelab`, 17 monitors, domain in Kuma |
+| **Kuma status page** | https://status.levkin.ca — slug `homelab`, 17 monitors |
 | **Umami** | https://stats.levkin.ca → `:3000` (Caddy + DNS) |
-| **Beszel** | 16 agents **up** (hub, pve10/pve201, P1 apps, vaultwarden, hermes, static LXCs) |
+| **Beszel** | 16 agents **up**; alerts Status/CPU/RAM/disk on all systems |
 | **SMTP** | `alerts@levkine.ca` — Kuma + Beszel; `listmonk@` — user UI + vault |

 ### Caddy (VM 106 @ `10.0.10.50`)

- `stats.levkin.ca` → `10.0.10.22:3000` (was already live)
- `status.levkin.ca` → `10.0.10.22:3001` (added 2026-05-24)
+- `stats.levkin.ca` → `10.0.10.22:3000`
+- `status.levkin.ca` → `10.0.10.22:3001`
 - Playbook: `playbooks/caddy-monitoring-sites.yml` · `make caddy-monitoring`

+### Phase 0 — Foundation ✅
+
+- All pve10 **LXCs** static via `pct set` (210, 215–221)
+- **VM 106** Caddy static in-guest `.50`
+- **UniFi DHCP** reservations applied for homelab VMs (API; key in vault)
+- **DNS** `auth.levkin.ca` + `levkin.ca` apex → home IP
+- **Identity LXC 217** @ `.21` provisioned (Phase 1 infra)
+
+Caddy still on **VM 106** — edge LXC → **Phase 2 backlog** ([handoff-next-steps.md](handoff-next-steps.md)).
+
 ### Infra / migrations

 - Nextcloud VM 201 retired; listmonk → LXC 221; pve201 VM 113 destroyed
 - Vikunja OIDC verified (browser login as `ilia`)
+- Listmonk upgraded **v2.4 → v6.1.0** + Authentik OIDC — [listmonk-authentik-oidc.md](listmonk-authentik-oidc.md)
 - Gitea deploy key levkin LXC 220 — added + tested
+- **DebianDesktop VM 100** (pve201) — rebooted; 24 GB RAM active
 - UniFi API key in vault; DHCP fixed IPs applied for homelab VMs

 ### Vault keys added/updated
@ -38,6 +57,15 @@

 ---

+## Capacity snapshot (2026-05-24)
+
+| Node | RAM available | Disk (local-lvm) | Notes |
+|------|---------------|------------------|--------|
+| **pve10** | ~22 GiB / 62 GiB | ~1.30 TiB free (~22% used) | Primary for new LXCs |
+| **pve201** | ~19 GiB / 125 GiB | ~922 GiB free (~46% used) | Do not add services |
+
+---
+
 ## Hostname map (monitoring)

 | DNS | Service | Backend | Public? |
@ -48,17 +76,15 @@
 | *(none)* | Kuma admin | `:3001` | LAN only — do not expose `/dashboard` |
 | *(none)* | Dockge | `:5001` | LAN only |

-**Hermes ≠ Mattermost:** VM **117** @ `.36` = Hermes agent; Mattermost = VM **107** @ `slack.levkin.ca` (`.237` in Caddy today).
+**Hermes ≠ Mattermost:** VM **117** @ `.36` = Hermes agent; Mattermost = VM **107** @ `slack.levkin.ca`.

 ---

-## Still open (manual / P1)
+## Sprint also delivered (Phases 2–4)

-1. **Beszel UI** — enable alerts per system (bell icon: Status, CPU, RAM, disk)
-2. **Rotate secrets** pasted in chat (UniFi key, Kuma password, Beszel admin if still temp)
-3. **Cal OIDC** — deferred (no `CALCOM_LICENSE_KEY`)
-4. **NAS / Jellyfin / DebianDesktop** — deferred
-5. **Optional:** `metrics.levkin.ca` → Beszel behind Tailscale/LAN-only Caddy block
+- Monitoring stack, Beszel alerts, SSO (Vikunja, Listmonk, Mattermost, Mailcow)
+
+**After merge:** see [handoff-next-steps.md](handoff-next-steps.md).

 ---

@ -69,16 +95,17 @@ make vault-export-env
 make caddy-monitoring          # stats + status on Caddy
 ./scripts/kuma-add-monitors.sh # needs KUMA_PASSWORD in .env
 ./scripts/beszel-setup-smtp.sh # Beszel SMTP via alerts@
+./scripts/beszel-setup-alerts.sh # Status + CPU/RAM/disk on all systems
 ssh root@10.0.10.22            # monitoring LXC
 ssh root@10.0.10.50            # Caddy VM
 ```

-## Key files touched
+## Key files

- `playbooks/caddy-monitoring-sites.yml` (new)
- `scripts/kuma-add-monitors.sh`, `scripts/beszel-setup-smtp.sh`
- `docs/guides/monitoring-stack.md`, `homelab-status-2026-05-22.md`, `levkin-selfhost-plan-2.md`
- `inventories/production/group_vars/all/vault.yml` (encrypted — do not commit plaintext)
+- `playbooks/caddy-monitoring-sites.yml`
+- `scripts/kuma-add-monitors.sh`, `scripts/beszel-setup-smtp.sh`, `scripts/beszel-setup-alerts.sh`
+- `docs/guides/monitoring-stack.md`, `levkin-selfhost-plan-2.md`
+- `inventories/production/group_vars/all/vault.yml` (encrypted)

 ## Beszel agent install quirk

--- a/docs/guides/handoff-next-steps.md
+++ b/docs/guides/handoff-next-steps.md
@ -0,0 +1,66 @@
+# Handoff — next steps (after post-sprint merge)
+
+**Merged from:** `homelab/post-sprint-2026-05-24` → `master`  
+**Sprint snapshot:** [handoff-2026-05-24.md](handoff-2026-05-24.md)  
+**Master plan:** [levkin-selfhost-plan-2.md](levkin-selfhost-plan-2.md)
+
+---
+
+## Phases complete
+
+| Phase | Status |
+|-------|--------|
+| **0 Foundation** | ✅ Static IPs, DNS, UniFi DHCP, Caddy VM `.50` |
+| **1 Identity** | ✅ Authentik LXC 217, `auth.levkin.ca`, admin + TOTP |
+| **2 Monitoring** | ✅ (sprint) Kuma, Umami, Beszel, Dockge, `status`/`stats` |
+| **3 Cal.com** | ✅ booking live; OIDC deferred (license) |
+| **4 SSO** | ✅ Vikunja, Listmonk, Mattermost, Mailcow — **smoke-test in browser** |
+
+**Not Phase 0/1:** Caddy → edge LXC `.20` moved to **Phase 2 backlog** (was Phase 1.5).
+
+---
+
+## Immediate (this week)
+
+1. **SSO smoke tests** (Playwright MCP or manual) as `ilia`:
+   - https://todo.levkin.ca — Authentik
+   - https://listmonk.levkin.ca/admin — Authentik
+   - https://slack.levkin.ca — “GitLab” / Authentik button
+   - https://mail.levkine.ca — Generic-OIDC
+2. **Rotate secrets** — Authentik API token, Beszel admin, OIDC client secrets (batch when stable)
+3. **Mattermost users** — existing accounts: Profile → Switch to GitLab SSO
+
+---
+
+## Phase 2 backlog (infra + ops)
+
+| Priority | Item | Effort |
+|----------|------|--------|
+| 1 | **Caddy → edge LXC** @ `10.0.10.20` | ~30 min + 24h watch |
+| 2 | **Security remediation** — [security-remediation-plan.md](security-remediation-plan.md) | ongoing |
+| 3 | **NAS disk** `W4J0L3PY` → Jellyfin VM 101 | hardware |
+| 4 | **Cal OIDC** | blocked on `CALCOM_LICENSE_KEY` |
+| 5 | **Phases 5–8** — Immich, Crater, Outline, etc. | when needed |
+
+---
+
+## Useful commands
+
+```bash
+make vault-export-env
+make caddy-monitoring
+make beszel-setup-alerts    # BESZEL_EMAIL + BESZEL_PASSWORD
+./scripts/kuma-add-monitors.sh
+ssh root@10.0.10.237          # Mattermost (root key installed)
+```
+
+## Docs added this sprint
+
+- [listmonk-authentik-oidc.md](listmonk-authentik-oidc.md)
+- [mattermost-authentik-gitlab-oauth.md](mattermost-authentik-gitlab-oauth.md)
+- [mailcow-authentik-oidc.md](mailcow-authentik-oidc.md)
+- [cursor-mcp-homelab.md](cursor-mcp-homelab.md)
+
+---
+
+*2026-05-24*
--- a/docs/guides/homelab-status-2026-05-22.md
+++ b/docs/guides/homelab-status-2026-05-22.md
@ -1,100 +0,0 @@
-# Homelab status — 2026-05-24
-
-Quick checklist. **Master plan:** [levkin-selfhost-plan-2.md](levkin-selfhost-plan-2.md)
-
-## Done (automation)
-
-| Item | Notes |
-|------|--------|
-| Mailcow `alerts@levkine.ca` | Created via API |
-| Kuma + Dockge + Umami | LXC 218 @ `10.0.10.22`; Dockge stack **monitoring** active |
-| **Kuma monitors** | **17** HTTP monitors; all linked to email alerts (`alerts@`) — [monitoring-stack.md](monitoring-stack.md#uptime-kuma--monitors) |
-| **Kuma admin password** | In vault (`vault_uptime_kuma_password`); `make vault-export-env` → `KUMA_PASSWORD` |
-| **Beszel hub + agents** | **16 systems all up** — hypervisors, P1 apps, vaultwarden, hermes, static LXCs — [monitoring-stack.md](monitoring-stack.md#4-beszel-agents-deployed) |
-| Old Kuma pve201 LXC 305 | Stopped, `onboot` off |
-| `stats.levkin.ca` | Caddy → Umami `:3000` |
-| Tracking scripts | levkin.ca + caseware + auto + portfolio (`iliadobkin.com`) |
-| **levkin.ca** | LXC **220** @ `10.0.10.60`; Caddy → nginx; `/` = spec, `/folders/` = stack |
-| Portfolio `iliadobkin.com` | Migrated pve201 LXC **306** → pve10 LXC **219** @ `10.0.10.106`; Caddy → nginx `:80` |
-| Kuma SMTP | Working (user confirmed) |
-| **DNS `levkin.ca` + `www`** | A → `142.180.237.136` verified 2026-05-23 |
-| **Nextcloud VM 201** | Stopped, `onboot 0`, Caddy block removed (~8 GiB RAM on pve10) |
-| **Listmonk** | pve201 VM **113** → pve10 LXC **221** @ `10.0.10.148`; SMTP `listmonk@` ✅ (user UI); pve201 VM **113 destroyed** |
-| **Phase 0 snapshots** | `pre-homelab-sprint-20260523` on pve10 217/218/210/106/201; pve201 106/113 + LXC 301 |
-| **Vikunja OIDC** | ✅ browser login as `ilia` verified 2026-05-24 — [vikunja-authentik-oidc.md](vikunja-authentik-oidc.md) |
-| Git remote | `git@git.levkin.ca:ilia/...` (SSH → `10.0.10.169` via `~/.ssh/config` on site LXCs) |
-| auto repo | Pushed/pulled on `git.levkin.ca` |
-| caseware repo | Pushed to Gitea via bundle on server; LXCs pull via internal SSH |
-| Vault | Mailcow, Umami, Mattermost in vault; `make vault-export-env` → `.env`; `make vault-pull-infra-secrets` = hosts → vault |
-| Caddy root SSH | Works (`make bootstrap-root-ssh-caddy`) |
-| Hermes agent (VM **117**) | @ `10.0.10.36` — Telegram bridge; **not** Mattermost |
-| Mattermost (VM **107**) | @ `slack.levkin.ca` — Kuma monitors this; separate from Hermes |
-| **CI / control docs** | [ci-runners-and-control.md](ci-runners-and-control.md) — git-ci-01, sonarqube-01, ansibleVM |
-| **SMTP inventory** | [smtp-inventory.md](smtp-inventory.md) — who uses which mailbox |
-
-## Capacity snapshot (2026-05-24)
-
-| Node | RAM available | Disk (local-lvm) | Notes |
-|------|---------------|------------------|--------|
-| **pve10** | ~22 GiB / 62 GiB | ~1.30 TiB free (~22% used) | Primary for new LXCs |
-| **pve201** | ~19 GiB / 125 GiB | ~922 GiB free (~46% used) | Do not add services |
-
-## Your list — still to do
-
-### You (UI / hardware / DNS)
-
- [x] **Kuma SMTP** — working
- [x] **DNS `levkin.ca` + `www`** — A records → `142.180.237.136`
- [x] **Gitea deploy key (levkin LXC 220)** — added + tested 2026-05-24
- [x] **UniFi DHCP** — API reservations applied (gitea, vaultwarden, n8n, hermes, actual, jellyfin, mailcow, truenas, PVE.BU.SVR); key in vault
- [x] **Beszel agents** — 16 systems (P1 + vaultwarden + hermes + statics) — [monitoring-stack.md](monitoring-stack.md#4-beszel-agents-deployed)
- [x] **Kuma monitors** — levkin.ca, portfolio, search, pdf, umami script, mattermost added
- [x] **Listmonk SMTP** — `listmonk@levkine.ca` updated in UI; vault synced
- [ ] **Cal.com → Authentik OIDC** — **deferred** (no license key)
- [x] **Portainer VM 109** — stopped and destroyed on pve10 (2026-05-23)
- [x] **Mailcow** — LAN TCP timeout fixed (netfilter `MAILCOW` drop rule) — [mailcow-lan-proxy-fix.md](mailcow-lan-proxy-fix.md)
- [ ] **DebianDesktop VM 100** — RAM lowered to 24 GB in Proxmox; **reboot guest** to apply balloon
- [x] **Nextcloud VM 201 retire** — Caddy removed, VM stopped, `onboot 0`
- [ ] **NAS.SP00 disk replace** — then start Jellyfin (VM 101) — **deferred (skip for P0)**
- [x] **Gitea deploy key (portfolio)** — `git pull` works on LXC 219; Gitea VM SSH fixed (`/home/git/.ssh/authorized_keys` + `sudo` to `gitea`)
- [x] **pve201 VM 113** — destroyed 2026-05-24 after listmonk LXC verified
- [ ] **Rotate** any secrets pasted in chat (Hermes token, etc.)
-
-### Next (manual / UI)
-
-1. **Beszel UI** — enable alerts per system (bell icon): Status (1 min), CPU/RAM/disk thresholds
-2. ~~**Vikunja**~~ — ✅ OIDC login as `ilia` verified
-3. ~~**UniFi DHCP**~~ — ✅ VM reservations via API (listmonk LXC 221 static via `pct set` — skip)
-4. ~~**Gitea deploy key**~~ — ✅ levkin LXC 220 tested
-5. ~~**Kuma status page**~~ — ✅ `status.levkin.ca` + Caddy
-
-**Deferred:** NAS SP00 / Jellyfin; DebianDesktop reboot; Cal OIDC (no license)
-
-### Later / defer
-
- [ ] Caddy → edge LXC `.20`
- [ ] Immich, Crater
- [ ] Public SSH for `git.levkin.ca:22` (optional Caddy `layer4` or DNS split)
-
-## Site LXCs (marketing)
-
-| VMID | Name | IP | Git remote |
-|------|------|-----|------------|
-| 220 | levkin | 10.0.10.60 | `git@git.levkin.ca:ilia/levkin.ca.git` |
-| 215 | caseware | 10.0.10.105 | `git@git.levkin.ca:ilia/caseware.git` |
-| 216 | auto | 10.0.10.59 | `git@git.levkin.ca:ilia/auto.git` |
-| 219 | portfolio | 10.0.10.106 | `git@git.levkin.ca:ilia/sdetProfile.git` |
-
-**Git SSH note:** `git.levkin.ca` in the URL; traffic goes to **10.0.10.169:22** (not `10.0.30.169`, not public `:22`).
-
-```ssh
-# On each site LXC /root/.ssh/config
-Host git.levkin.ca
-    HostName 10.0.10.169
-    User git
-    IdentityFile ~/.ssh/id_ed25519
-```
-
-## Dockge
-
-Stack **monitoring** in UI = correct. Compose at `/opt/stacks/monitoring/compose.yaml`. Live stack also at `/opt/monitoring` (same containers). Use Dockge for edits/restarts; avoid starting a second copy.
--- a/docs/guides/host-list.md
+++ b/docs/guides/host-list.md
@ -1,7 +1,7 @@
 # Host list — Proxmox guests (source of truth)

 **Node:** PVENAS (`pve10` @ `10.0.10.10`)  
-**Audited:** 2026-05-22 (Phase 0 IP pass + monitoring LXC 218 provisioned)  
+**Audited:** 2026-05-24 (Phase 0 complete — LXCs static + UniFi VM DHCP)  
 **LAN:** `10.0.10.0/24`, gateway `10.0.10.1`

 Update this file whenever a guest is created, migrated, or re-IP’d. See [levkin-selfhost-plan-2.md](levkin-selfhost-plan-2.md) for IP range policy.
@ -61,7 +61,7 @@ Update this file whenever a guest is created, migrated, or re-IP’d. See [levki
 | 104 | vaultwarden-debian | identity | `10.0.10.142/24` | `10.0.10.142/24` | ⏳ stable DHCP | `BC:24:11:58:DB:DC` | Inventory `vaultwardenVM` |
 | 105 | TrueNAS | — | `10.0.10.107/24` | `10.0.10.107/24` | ⏳ stable DHCP | `BC:24:11:14:DE:B5` | NAS UI; pool `NAS.SP00` degraded |
 | 106 | caddy-debian | **edge** | `10.0.10.50/24` | `10.0.10.50/24` → **`.20`** (Phase 1.5) | ✅ **Static** (in-guest) | `BC:24:11:E0:49:B4` | `/etc/network/interfaces` static; Ansible `caddy` |
-| 107 | mattermost-ubuntu | comms | `10.0.10.107`? | TBD | ⏳ | `BC:24:11:66:6E:01` | Ping `.107` up; confirm not TrueNAS conflict — verify in guest |
+| 107 | mattermost-ubuntu | comms | `10.0.10.237/24` | `10.0.10.237/24` | ⏳ router DHCP | `BC:24:11:66:6E:01` | `slack.levkin.ca` → Caddy → `:8065` |
 | 108 | actual-debian | business | `10.0.10.158/24` | `10.0.10.158/24` | ⏳ stable DHCP | `BC:24:11:10:7B:64` | Inventory `actual` |
 | 109 | portainer-alpine | — | — | — | ✅ **Removed** | `BC:24:11:0F:40:4F` | Destroyed 2026-05-23; Dockge on monitoring LXC 218 |
 | 150 | pihole00-debian | — | link-local* | TBD | ⏳ | `BC:24:11:86:76:97` | Running |
@ -107,7 +107,7 @@ Priority order (plan-2):
 3. ✅ **LXC 217** (identity) — `10.0.10.21/24`, Authentik deployed
 4. ✅ **VM 106** (caddy) — static in-guest `.50`
 5. ✅ **LXC 218** (monitoring) — `.22`, Kuma/Dockge/Umami
-6. **VMs** — use [vm-static-ip-router-reservations.md](vm-static-ip-router-reservations.md) (router MAC reservations); skip **201** (Nextcloud retire)
+6. ✅ **VMs** — UniFi DHCP reservations applied 2026-05-24 — [vm-static-ip-router-reservations.md](vm-static-ip-router-reservations.md); skip **201** (retired)
 7. **New:** edge LXC @ **`.20`** (Phase 1.5)

 Example:
@ -150,8 +150,8 @@ See [ci-runners-and-control.md](ci-runners-and-control.md).
 - [x] Monitoring LXC 218 @ `.22`
 - [x] Caddy VM 106 static `.50`
 - [x] LXC backups `backup-20260522` on 217, 218
- [ ] Router DHCP reservations for VMs — [vm-static-ip-router-reservations.md](vm-static-ip-router-reservations.md) (manual in router UI; table ready)
+- [x] Router DHCP reservations for VMs — UniFi API 2026-05-24
 - [x] Retire VM 201 (Nextcloud) — stopped 2026-05-23
- [x] Listmonk → pve10 LXC 221 @ `.148`
- [ ] UniFi: update listmonk DHCP MAC `BC:24:11:18:0C:62` (was VM 113 `…:11:53:9A`)
+- [x] Listmonk → pve10 LXC 221 @ `.148` (static via `pct set`; no UniFi lease needed)
+- [x] Phase 0 complete — all critical guests pinned
 - [ ] Re-run after NAS disk replace
--- a/docs/guides/levkin-selfhost-plan-2.md
+++ b/docs/guides/levkin-selfhost-plan-2.md
@ -11,15 +11,15 @@ Reference doc for the Proxmox homelab. Lives alongside the Cursor project that h

 ---

-## Progress summary (updated 2026-05-23)
+## Progress summary (updated 2026-05-24)

 | Area | Status |
 |------|--------|
-| **Phase 0** Foundation | ✅ Mostly done — pve10 LXCs static; site LXCs 215/216/219/220 static; Caddy still on **VM 106** @ `.50` |
+| **Phase 0** Foundation | ✅ **Done** — pve10 LXCs static; UniFi VM DHCP reservations; auth + apex DNS; Caddy on **VM 106** @ `.50` (edge LXC = Phase 1.5) |
 | **Phase 1** Identity (Authentik) | ✅ LXC **217** @ `10.0.10.21` — admin + TOTP |
 | **Phase 2** Monitoring | ✅ LXC **218** — Kuma (17 monitors), Dockge, Umami, Beszel (16 agents), SMTP |
 | **Phase 3** Cal.com | ✅ LXC **210** — booking + auto consult button; **OIDC deferred** (no enterprise license) |
-| **Phase 4** SSO | ⏳ **Next:** Vikunja → Authentik — [sso-selfhosted-matrix.md](sso-selfhosted-matrix.md) |
+| **Phase 4** SSO | ✅ Vikunja, Listmonk, Mattermost, Mailcow — browser smoke tests remaining |
 | **Phase 5–8** | ⏳ Immich, Crater, Outline, automation depth — after P0 backlog |
 | **Comms health** | ✅ Mailcow + Listmonk restored 2026-05-23 — [mailcow-lan-proxy-fix.md](mailcow-lan-proxy-fix.md) |
 | **Site consolidation** | ⏳ **Partial** — git LXCs + levkin.ca LXC 220; optional later: static on Caddy VM |
@ -59,7 +59,7 @@ Use this before adding LXCs/VMs. Re-check with `pvesm status` and `free -h` on e
 | Resource | Total | Used | **Available** | Notes |
 |----------|-------|------|---------------|--------|
 | **local-lvm** | ~1.67 TiB | ~46% | **~922 GiB** | Disk OK |
-| **RAM** | 125 GiB | ~105 GiB | **~19 GiB** | GPU **104** (64 GB), DebianDesktop **100** (reboot for 24 GB), punim **9101** (16 GB) |
+| **RAM** | 125 GiB | ~105 GiB | **~19 GiB** | GPU **104** (64 GB), DebianDesktop **100** (24 GB ✅ rebooted), punim **9101** (16 GB) |

 **Verdict:** New stacks on **pve10** only. pve201: stop/migrate punim after testing.

@ -70,7 +70,7 @@ Use this before adding LXCs/VMs. Re-check with `pvesm status` and `free -h` on e
 **Already running:**
 - Caddy reverse proxy — currently on a **VM** (should migrate to LXC, see "Caddy migration" section)
 - Mailcow — VM, mail domain is `levkine.ca` (with e)
- Vaultwarden, Vikunja, n8n, Listmonk, Mattermost, Nextcloud — across various LXCs
+- Vaultwarden, Vikunja, n8n, Listmonk, Mattermost — across various LXCs/VMs
 - **Cal.com** — LXC id `210`, `cal.levkin.ca`, Postgres included, admin user `ilia`, 15-min consult event live at `cal.levkin.ca/ilia/consult` with Jitsi link
 - Caddy entries live for: `levkin.ca`, `caseware.levkin.ca`, `auto.levkin.ca`, `iliadobkin.com`, `cal.levkin.ca`, `listmonk.levkin.ca`, `pdf.levkin.ca`, `search.levkin.ca`, `auth.levkin.ca`, `stats.levkin.ca`, **`status.levkin.ca`**
 - **Authentik** — LXC **217** @ `10.0.10.21`, `https://auth.levkin.ca`, admin + TOTP enrolled
@ -80,8 +80,8 @@ Use this before adding LXCs/VMs. Re-check with `pvesm status` and `free -h` on e
 - **Dockge** on 218 — manages local `/opt/monitoring` stack
 - **Snapshots** `backup-20260522` on LXCs **217**, **218**
 - **Jellyfin** (VM 101) — stopped
- LXC **210, 215–218, 219** — static via `pct set`; **Caddy VM 106** — static in-guest `.50`
- **Nextcloud VM 201** — export done; VM **still running** on pve10 — **retire next** (8 GB RAM reclaimed)
+- LXC **210, 215–221** — static via `pct set`; **Caddy VM 106** — static in-guest `.50`
+- **Nextcloud VM 201** — retired (stopped, `onboot 0`, Caddy removed)
 - ~~**Portainer VM 109**~~ — **removed** 2026-05-23 (~16 GiB RAM freed on pve10)
 - **Marketing sites** — LXC **220** (`levkin.ca`), **215/216/219** (git deploy), not yet on Caddy VM static roots
 - **punimTag dev** — pve201 LXC **9101** @ `10.0.10.121` (16 GB) — leave until testing done; then `dev-apps` on pve10
@ -174,10 +174,10 @@ Reachable only via local network or Tailscale/Wireguard:

 ## Phased rollout

-### Phase 0 — Foundation
+### Phase 0 — Foundation ✅
 1. ✅ Caddy running (on VM — migrate to LXC in Phase 1.5)
-2. ✅ **Static IP audit (partial)** — all LXCs on pve10 pinned; Caddy VM static `.50`; remaining VMs on stable DHCP — see [host-list.md](host-list.md)
-3. ✅ DNS for `auth.levkin.ca` → home IP (verified 2026-05-22)
+2. ✅ **Static IP audit** — all pve10 LXCs pinned via `pct set`; Caddy VM static `.50`; homelab VMs pinned via UniFi DHCP — see [host-list.md](host-list.md)
+3. ✅ DNS for `auth.levkin.ca` + `levkin.ca` apex → home IP
 4. ✅ `identity` LXC **217** @ `10.0.10.21` (2 vCPU, 2GB RAM, 20GB `local-lvm`, Debian 12 + Docker Compose)

 ### Phase 1 — Identity ✅
@ -187,9 +187,9 @@ Reachable only via local network or Tailscale/Wireguard:
 4. ✅ `authentik Admins` group (skip custom `users` group until more accounts)
 5. ✅ Static backup codes; **don't OIDC other apps until Cal.com test**

-### Phase 1.5 — Caddy migration to LXC (~30 min)
+### Phase 2 — Next infra (was Phase 1.5) — Caddy migration to LXC ⏳

-Why now (after Phase 1, before bulk SSO work in Phase 4): Authentik is stable enough to absorb a small change, but you haven't yet built the dependency web of OIDC integrations that would make a Caddy reload risky.
+Deferred until after sprint merge. Authentik + SSO are stable; edge migration is the next structural change.

 Why Caddy belongs in an LXC, not a VM:
 - ~50MB OS overhead vs ~512MB for a VM
@ -217,7 +217,7 @@ Steps:
 1. ✅ **Umami** — tracking on levkin.ca, caseware, auto, and iliadobkin.com (portfolio)
 2. ✅ **Uptime Kuma** — monitors in UI
 3. ✅ **Dockge** — logged in; register `/opt/monitoring` stack (see [monitoring-stack.md](monitoring-stack.md))
-4. ✅ **Kuma email alerts** — SMTP via Mailcow (see [homelab-status-2026-05-22.md](homelab-status-2026-05-22.md))
+4. ✅ **Kuma email alerts** — SMTP via Mailcow — [monitoring-stack.md](monitoring-stack.md)

 ### Phase 3 — Cal.com (mostly done) ✅
 1. ✅ Cal.com deployed in `business` LXC (id 210, Postgres included)
@ -227,13 +227,14 @@ Steps:
 5. ⏳ **Cal.com OIDC** — **deferred** ([cal-authentik-oidc.md](cal-authentik-oidc.md)) — needs enterprise `CALCOM_LICENSE_KEY`
 6. ✅ `auto.levkin.ca` consult button → `cal.levkin.ca/ilia/consult`

-### Phase 4 — SSO migration (~half a day, staged)
-Wire each to Authentik, least-risky first:
-1. **Vikunja** (OIDC native) — easy, single-user impact
-2. ~~**Nextcloud**~~ — **skipped** (VM 201 retiring)
-3. **Listmonk** (OIDC native, admin only) — easy
-4. **Mattermost** (SAML or OIDC native) — moderate
-5. **Mailcow** (OIDC) — last, because mail-critical
+### Phase 4 — SSO migration ✅
+1. ✅ **Vikunja** — [vikunja-authentik-oidc.md](vikunja-authentik-oidc.md)
+2. ~~**Nextcloud**~~ — skipped (VM 201 retired)
+3. ✅ **Listmonk** — [listmonk-authentik-oidc.md](listmonk-authentik-oidc.md) (v6.1.0)
+4. ✅ **Mattermost** — [mattermost-authentik-gitlab-oauth.md](mattermost-authentik-gitlab-oauth.md)
+5. ✅ **Mailcow** — [mailcow-authentik-oidc.md](mailcow-authentik-oidc.md)
+
+**Remaining:** browser smoke tests as `ilia`; rotate OIDC secrets when done.

 For each: keep a local admin password as a break-glass account.

@ -338,13 +339,15 @@ pct reboot <ID>
 | 10 | NAS / Jellyfin / DebianDesktop | **deferred** |
 | 11 | Cal OIDC | deferred (no license) |

-### P1 — next (manual / UI)
+### P1 — next

-1. **Beszel alerts** — bell icon per system in hub UI
-2. **Vikunja** — browser login as `ilia`
-3. **UniFi** — listmonk MAC `BC:24:11:18:0C:62`
-4. **Gitea deploy key** — levkin LXC 220
-5. **Kuma status page** — optional `status.levkin.ca`
+See **[handoff-next-steps.md](handoff-next-steps.md)** — SSO smoke tests, secret rotation.
+
+### Phase 2 backlog (was P1 infra)
+
+1. **Caddy → edge LXC** @ `10.0.10.20`
+2. **Security remediation** — [security-remediation-plan.md](security-remediation-plan.md)
+3. **NAS / Jellyfin** — disk `W4J0L3PY`

 ### P1 — when ready
 - **Outline** — wiki for client docs
@ -388,7 +391,7 @@ The whole LXC gets snapshotted — much simpler than file-level container backup

 ## Next steps (priority order)

-See **[homelab-status-2026-05-22.md](homelab-status-2026-05-22.md)** for automation checklist.
+See **[handoff-2026-05-24.md](handoff-2026-05-24.md)** for sprint status checklist.

 | # | Task | Status | Effort | Frees / unlocks |
 |---|------|--------|--------|-----------------|
@ -403,7 +406,7 @@ See **[homelab-status-2026-05-22.md](homelab-status-2026-05-22.md)** for automat
 | 9 | **Beszel + Kuma** | ✅ | — | 16 Beszel agents; 17 Kuma monitors |
 | 10 | ~~**Listmonk SMTP**~~ | ✅ | — | UI + vault |
 | 10 | **NAS.SP00** disk → Jellyfin | ⏳ hardware | — | VM 101 |
-| 11 | **DebianDesktop reboot** | ⏳ | 5 min | Apply 24 GB on pve201 |
+| 11 | **DebianDesktop reboot** | ✅ | — | VM 100 rebooted; 24 GB active on pve201 |
 | 12 | **Caddy → edge LXC `.20`** | ⏳ defer | ~30 min | Phase 1.5 |
 | 13 | **dev-apps LXC** | ⏳ defer | half day | After punim testing |
 | 14 | **Static sites → Caddy VM** | ⏳ optional | 1 h | Defer |
--- a/docs/guides/listmonk-authentik-oidc.md
+++ b/docs/guides/listmonk-authentik-oidc.md
@ -0,0 +1,50 @@
+# Listmonk ↔ Authentik OIDC
+
+**Status:** Live at `https://listmonk.levkin.ca` (LXC **221**, `10.0.10.148`).
+
+**Requires listmonk v5+** (OIDC). Upgraded from v2.4.0 → **v6.1.0** on 2026-05-24.
+
+## Authentik
+
+| Item | Value |
+|------|--------|
+| Application slug | `listmonk` |
+| Provider name | `listmonk-oidc` |
+| Client ID | `listmonk` |
+| Redirect URI (strict) | `https://listmonk.levkin.ca/auth/oidc` |
+| Subject mode | **user_username** |
+| Signing key | `authentik Self-signed Certificate` |
+| Access group | **`homelab-users`** (app binding) |
+
+Client secret: `vault_listmonk_oidc_client_secret` in Ansible vault (rotate if exposed).
+
+## Listmonk
+
+Configured via **Settings → Security → OIDC** (stored in DB):
+
+- **Provider URL:** `https://auth.levkin.ca/application/o/listmonk/`
+- **Auto-create users:** enabled (Super Admin role id `1` for new SSO users)
+
+Break-glass: local user `listmonk` (password login still enabled).
+
+## Login
+
+1. Sign out: `https://auth.levkin.ca/if/user/logout/`
+2. `https://listmonk.levkin.ca/admin` → **Login with Authentik**
+3. Sign in as **`ilia`** (must be in `homelab-users`)
+
+## Upgrade (if needed)
+
+```bash
+ssh root@10.0.10.148
+systemctl stop listmonk
+curl -fsSL -o /tmp/lm.tgz https://github.com/knadh/listmonk/releases/download/v6.1.0/listmonk_6.1.0_linux_amd64.tar.gz
+tar -xzf /tmp/lm.tgz -C /tmp && mv /tmp/listmonk /root/listmonk
+/root/listmonk --config /etc/listmonk/config.toml --upgrade --yes
+systemctl start listmonk
+```
+
+## Related
+
+- [sso-selfhosted-matrix.md](sso-selfhosted-matrix.md)
+- [Listmonk OIDC docs](https://listmonk.app/docs/oidc/)
--- a/docs/guides/mailcow-authentik-oidc.md
+++ b/docs/guides/mailcow-authentik-oidc.md
@ -0,0 +1,52 @@
+# Mailcow ↔ Authentik OIDC
+
+**Status:** Configured 2026-05-24 (Generic-OIDC in DB + Authentik app `mailcow`)
+
+**Requires:** mailcow **2025-03+** (this host: `2025-10a`)
+
+**URL:** https://mail.levkine.ca
+
+---
+
+## What OIDC means
+
+**OIDC** = **OpenID Connect** — login with an identity provider (Authentik) instead of a separate password per app. You sign in once at `auth.levkin.ca`, apps trust that login.
+
+---
+
+## Authentik
+
+| Item | Value |
+|------|--------|
+| Application slug | `mailcow` |
+| Provider | `mailcow-oidc` |
+| Client ID | `mailcow` |
+| Redirect URI | `https://mail.levkine.ca` |
+| Scope mapping | `mailcow_template` → `default` mailbox template |
+| Access | `homelab-users` |
+
+Secret: `vault_mailcow_oidc_client_secret` in Ansible vault.
+
+---
+
+## Mailcow (applied via MySQL `identity_provider`)
+
+- **Identity Provider:** Generic-OIDC
+- **Authorize / token / userinfo:** `https://auth.levkin.ca/application/o/{authorize,token,userinfo}/`
+- **Redirect URL:** `https://mail.levkine.ca`
+- **Scopes:** `openid profile email mailcow_template`
+
+Mailbox users with SSO need matching email in Authentik. Admin UI may still use local admin for break-glass.
+
+---
+
+## Verify
+
+Log out of Mailcow → login should offer external IdP. Test with user `ilia` in `homelab-users`.
+
+---
+
+## Related
+
+- [sso-selfhosted-matrix.md](sso-selfhosted-matrix.md)
+- [Authentik mailcow integration](https://integrations.goauthentik.io/chat-communication-collaboration/mailcow/)
--- a/docs/guides/mattermost-authentik-gitlab-oauth.md
+++ b/docs/guides/mattermost-authentik-gitlab-oauth.md
@ -0,0 +1,74 @@
+# Mattermost ↔ Authentik (GitLab OAuth workaround)
+
+**Status:** ✅ Live (config.json patched 2026-05-24; VM **107** @ `10.0.10.237`)
+
+Team Edition has no generic OIDC UI — use **GitLab OAuth** endpoints pointed at Authentik.
+
+**URL:** https://slack.levkin.ca · **Backend:** `10.0.10.237:8065` (VM **107** on pve10)
+
+---
+
+## Authentik (done 2026-05-24)
+
+| Item | Value |
+|------|--------|
+| Application slug | `mattermost` |
+| Provider | `mattermost-gitlab-oidc` |
+| Client ID | `mattermost` |
+| Redirect URI | `https://slack.levkin.ca/signup/gitlab/complete` |
+| Scope mappings | `mattermost-username`, `mattermost-id` + default OpenID |
+| Access | `homelab-users` group binding |
+
+Client secret: store in vault as `vault_mattermost_oidc_client_secret` (rotate if exposed).
+
+---
+
+## Mattermost — apply on VM
+
+SSH as root (or bootstrap key first: `make bootstrap-root-ssh` once password works):
+
+```bash
+ssh root@10.0.10.237
+```
+
+Edit `/opt/mattermost/config/config.json` (path may vary — `find / -name config.json -path '*mattermost*'`).
+
+Set `GitLabSettings`:
+
+```json
+"GitLabSettings": {
+  "Enable": true,
+  "Secret": "<vault_mattermost_oidc_client_secret>",
+  "Id": "mattermost",
+  "Scope": "",
+  "AuthEndpoint": "https://auth.levkin.ca/application/o/authorize/",
+  "TokenEndpoint": "https://auth.levkin.ca/application/o/token/",
+  "UserAPIEndpoint": "https://auth.levkin.ca/application/o/userinfo/",
+  "DiscoveryEndpoint": "https://auth.levkin.ca/application/o/mattermost/.well-known/openid-configuration",
+  "ButtonText": "Log in with Authentik",
+  "ButtonColor": "#fd4b2d"
+}
+```
+
+Then:
+
+1. **System Console** → Authentication → Signup → **Enable Account Creation** = true  
+2. `systemctl restart mattermost` (or `docker compose restart` if containerized)  
+3. Log out → use **GitLab** button (actually Authentik)  
+4. Existing users: Profile → Security → **Switch to GitLab SSO** (see [Authentik integration](https://integrations.goauthentik.io/chat-communication-collaboration/mattermost-team-edition/))
+
+---
+
+## Verify
+
+```bash
+curl -sS https://auth.levkin.ca/application/o/mattermost/.well-known/openid-configuration | head
+curl -sS -o /dev/null -w '%{http_code}\n' https://slack.levkin.ca/login
+```
+
+---
+
+## Related
+
+- [sso-selfhosted-matrix.md](sso-selfhosted-matrix.md)
+- [cursor-mcp-homelab.md](cursor-mcp-homelab.md) — Playwright can smoke-test login after `config.json` is applied
--- a/docs/guides/monitoring-stack.md
+++ b/docs/guides/monitoring-stack.md
@ -54,6 +54,7 @@ After save: `ssh root@10.0.10.22 'cd /opt/monitoring && docker compose restart b
 ```bash
 export BESZEL_EMAIL='you@example.com' BESZEL_PASSWORD='...' SMTP_PASS='...'
 ./scripts/beszel-setup-smtp.sh
+make beszel-setup-alerts   # Status + CPU/RAM/disk on all systems
 ```

 ### 3. Local agent (monitoring LXC)
--- a/docs/guides/security-remediation-plan.md
+++ b/docs/guides/security-remediation-plan.md
@ -449,7 +449,7 @@ Also tracked in [security-audit-report.md](security-audit-report.md) remediation
 - [ ] 8006 restricted to admin subnet/IP
 - [x] SSH keys on most inventory hosts (2026-05-23 — see audit report)
 - [ ] SSH keys on **caddy**, **ansibleVM**, **vaultwardenVM** (if still pending)
- [x] pve201 RAM partial relief — GPU 64 GiB; DebianDesktop 24 GiB (**reboot guest**)
+- [x] pve201 RAM partial relief — GPU 64 GiB; DebianDesktop 24 GiB (**VM 100 rebooted 2026-05-24**)

 ### High

--- a/docs/guides/smtp-inventory.md
+++ b/docs/guides/smtp-inventory.md
@ -39,6 +39,7 @@
 ```bash
 make vault-export-env
 ./scripts/beszel-setup-smtp.sh   # BESZEL_EMAIL, BESZEL_PASSWORD, SMTP_PASS
+./scripts/beszel-setup-alerts.sh # BESZEL_EMAIL, BESZEL_PASSWORD
 ```

 ---
--- a/docs/guides/sso-selfhosted-matrix.md
+++ b/docs/guides/sso-selfhosted-matrix.md
@ -32,9 +32,9 @@ Wire these first — typical OSS OIDC, no extra license:
 | App | OIDC/SAML | Notes |
 |-----|-----------|--------|
 | **Vikunja** | OIDC native | **Live** — [vikunja-authentik-oidc.md](vikunja-authentik-oidc.md); group `homelab-users` |
-| **Listmonk** | OIDC native | Admin-only |
-| **Mattermost** | OIDC or SAML | Moderate |
-| **Mailcow** | OIDC | Last — mail-critical |
+| **Listmonk** | OIDC native | **Live** — [listmonk-authentik-oidc.md](listmonk-authentik-oidc.md); v6.1.0+ |
+| **Mattermost** | GitLab OAuth → Authentik | ✅ [mattermost-authentik-gitlab-oauth.md](mattermost-authentik-gitlab-oauth.md) |
+| **Mailcow** | Generic-OIDC | ✅ [mailcow-authentik-oidc.md](mailcow-authentik-oidc.md) — test mailbox login |
 | **Umami** | — | Already LAN-only; no SSO needed |
 | **Vaultwarden** | — | **Do not OIDC** (break-glass) |
 | **n8n** | OIDC (if enabled) | Check edition |
--- a/inventories/production/group_vars/all/vault.example.yml
+++ b/inventories/production/group_vars/all/vault.example.yml
@ -62,7 +62,23 @@ vault_cal_oidc_client_secret: "CHANGE_ME"
 vault_vikunja_oidc_client_id: "CHANGE_ME"
 vault_vikunja_oidc_client_secret: "CHANGE_ME"

-# Hermes Mattermost (not Telegram)
+# Listmonk ↔ Authentik OIDC
+vault_listmonk_oidc_client_id: "listmonk"
+vault_listmonk_oidc_client_secret: "CHANGE_ME"
+
+# Authentik API (Integrations → Tokens) — for automation scripts
+vault_authentik_url: "https://auth.levkin.ca"
+vault_authentik_api_token: "CHANGE_ME"
+
+# Beszel hub admin (monitoring LXC :8090)
+vault_beszel_email: "CHANGE_ME"
+vault_beszel_password: "CHANGE_ME"
+
+# Mattermost Team Edition @ slack.levkin.ca (GitLab OAuth → Authentik)
+vault_mattermost_oidc_client_id: "mattermost"
+vault_mattermost_oidc_client_secret: "CHANGE_ME"
+
+# Hermes bridge (VM 117 — not Mattermost server)
 vault_mattermost_url: "https://slack.levkin.ca"
 vault_mattermost_token: "CHANGE_ME"
 vault_mattermost_allowed_users: "CHANGE_ME"
--- a/inventories/production/hosts
+++ b/inventories/production/hosts
@ -37,6 +37,7 @@ ansibleVM ansible_host=10.0.10.157 ansible_user=master
 # pve201 — email + newsletters
 mailcow ansible_host=10.0.10.132 ansible_user=root url=https://mail.levkine.ca proxmox_vmid=106 proxmox_node=pve201
 listmonk ansible_host=10.0.10.148 ansible_user=root url=https://listmonk.levkin.ca proxmox_vmid=221 proxmox_node=PVENAS
+mattermost ansible_host=10.0.10.237 ansible_user=root url=https://slack.levkin.ca proxmox_vmid=107 proxmox_node=PVENAS

 [services]
 # VMID 117: on PVENAS (pve10)
--- a/scripts/beszel-setup-alerts.sh
+++ b/scripts/beszel-setup-alerts.sh
@ -0,0 +1,132 @@
+#!/usr/bin/env bash
+# Enable Beszel alerts (Status, CPU, Memory, Disk) on all monitored systems.
+#
+# Prerequisite: admin account + SMTP configured (./scripts/beszel-setup-smtp.sh)
+#
+# Usage:
+#   export BESZEL_URL=http://10.0.10.22:8090
+#   export BESZEL_EMAIL=you@example.com
+#   export BESZEL_PASSWORD='your-beszel-password'
+#   ./scripts/beszel-setup-alerts.sh
+#
+# Optional thresholds (percent unless noted):
+#   BESZEL_CPU_THRESHOLD=80
+#   BESZEL_MEM_THRESHOLD=85
+#   BESZEL_DISK_THRESHOLD=90
+
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+BESZEL_URL="${BESZEL_URL:-http://10.0.10.22:8090}"
+BESZEL_EMAIL="${BESZEL_EMAIL:-}"
+BESZEL_PASSWORD="${BESZEL_PASSWORD:-}"
+BESZEL_CPU_THRESHOLD="${BESZEL_CPU_THRESHOLD:-80}"
+BESZEL_MEM_THRESHOLD="${BESZEL_MEM_THRESHOLD:-85}"
+BESZEL_DISK_THRESHOLD="${BESZEL_DISK_THRESHOLD:-90}"
+
+if [[ -z "${BESZEL_EMAIL}" || -z "${BESZEL_PASSWORD}" ]] && [[ -f "${REPO_ROOT}/.env" ]]; then
+  # shellcheck disable=SC1091
+  set +u
+  set -a
+  # shellcheck source=/dev/null
+  source "${REPO_ROOT}/.env"
+  set +a
+  set -u
+  BESZEL_EMAIL="${BESZEL_EMAIL:-}"
+  BESZEL_PASSWORD="${BESZEL_PASSWORD:-}"
+fi
+
+if [[ -z "${BESZEL_EMAIL}" || -z "${BESZEL_PASSWORD}" ]]; then
+  echo "Set BESZEL_EMAIL and BESZEL_PASSWORD (Beszel admin)" >&2
+  exit 1
+fi
+
+export BESZEL_URL BESZEL_EMAIL BESZEL_PASSWORD
+export BESZEL_CPU_THRESHOLD BESZEL_MEM_THRESHOLD BESZEL_DISK_THRESHOLD
+
+"${REPO_ROOT}/.venv/bin/python3" <<'PY'
+import json
+import os
+import sys
+import urllib.error
+import urllib.parse
+import urllib.request
+
+base = os.environ["BESZEL_URL"].rstrip("/")
+email = os.environ["BESZEL_EMAIL"]
+password = os.environ["BESZEL_PASSWORD"]
+cpu_threshold = float(os.environ["BESZEL_CPU_THRESHOLD"])
+mem_threshold = float(os.environ["BESZEL_MEM_THRESHOLD"])
+disk_threshold = float(os.environ["BESZEL_DISK_THRESHOLD"])
+
+ALERTS = [
+    ("Status", None),
+    ("CPU", cpu_threshold),
+    ("Memory", mem_threshold),
+    ("Disk", disk_threshold),
+]
+
+
+def req(method, path, token=None, body=None):
+    url = f"{base}{path}"
+    data = None
+    headers = {"Content-Type": "application/json"}
+    if body is not None:
+        data = json.dumps(body).encode()
+    if token:
+        headers["Authorization"] = token
+    request = urllib.request.Request(url, data=data, headers=headers, method=method)
+    try:
+        with urllib.request.urlopen(request, timeout=30) as resp:
+            raw = resp.read().decode()
+            return resp.status, json.loads(raw) if raw else {}
+    except urllib.error.HTTPError as e:
+        raw = e.read().decode()
+        print(f"HTTP {e.code} {path}: {raw}", file=sys.stderr)
+        raise
+
+
+print(f"Login to Beszel at {base} as {email}...")
+token = None
+for collection in ("_superusers", "users"):
+    try:
+        _, auth = req(
+            "POST",
+            f"/api/collections/{collection}/auth-with-password",
+            body={"identity": email, "password": password},
+        )
+        token = auth.get("token")
+        if token:
+            print(f"Login OK ({collection})")
+            break
+    except urllib.error.HTTPError:
+        continue
+
+if not token:
+    print("Login failed: check BESZEL_EMAIL and BESZEL_PASSWORD", file=sys.stderr)
+    sys.exit(1)
+
+query = urllib.parse.urlencode({"perPage": 500, "fields": "id,name"})
+_, systems_resp = req("GET", f"/api/collections/systems/records?{query}", token=token)
+systems = systems_resp.get("items", [])
+if not systems:
+    print("No systems found in Beszel hub", file=sys.stderr)
+    sys.exit(1)
+
+system_ids = [s["id"] for s in systems]
+names = ", ".join(s.get("name", s["id"]) for s in systems)
+print(f"Found {len(system_ids)} systems: {names}")
+
+for alert_name, threshold in ALERTS:
+    body = {"name": alert_name, "systems": system_ids}
+    if threshold is not None:
+        body["value"] = threshold
+    req("POST", "/api/beszel/user-alerts", token=token, body=body)
+    if threshold is None:
+        print(f"  ✅ {alert_name} (down detection)")
+    else:
+        print(f"  ✅ {alert_name} > {threshold}%")
+
+print("\nDone. Alerts apply to all systems. Verify in hub UI (bell icon) or send test:")
+print(f"  curl -X POST {base}/api/beszel/test-notification -H 'Authorization: {token[:20]}...'")
+PY
--- a/scripts/beszel-setup-smtp.sh
+++ b/scripts/beszel-setup-smtp.sh
@ -24,11 +24,14 @@ SMTP_USER="${SMTP_USER:-alerts@levkine.ca}"
 SMTP_PASS="${SMTP_PASS:-}"
 SMTP_TO="${SMTP_TO:-idobkin@gmail.com}"

-if [[ -f "${REPO_ROOT}/.env" ]]; then
+if [[ -z "${SMTP_PASS}" ]] && [[ -f "${REPO_ROOT}/.env" ]]; then
  # shellcheck disable=SC1091
+  set +u
  set -a
+  # shellcheck source=/dev/null
  source "${REPO_ROOT}/.env"
  set +a
+  set -u
  SMTP_PASS="${SMTP_PASS:-${ALERTS_PASSWORD:-}}"
 fi