Merge pull request 'Homelab sprint: monitoring stack, Caddy public URLs, vault secrets.' (#10) from homelab-sprint-2026-05-23 into master

Reviewed-on: #10
2026-05-24 11:11:34 -05:00 · 2026-05-24 11:11:34 -05:00 · b3f2d438e5
commit b3f2d438e5
parent d121663765 9c271792a5
19 changed files with 926 additions and 222 deletions
--- a/8
+++ b/8
@ -707,6 +707,14 @@ kuma-add-monitors: ## Add default Uptime Kuma monitors (needs KUMA_PASSWORD in .
 	@chmod +x scripts/kuma-add-monitors.sh
 	@./scripts/kuma-add-monitors.sh

+beszel-setup-smtp: ## Configure Beszel SMTP (needs BESZEL_EMAIL, BESZEL_PASSWORD, SMTP_PASS)
+	@chmod +x scripts/beszel-setup-smtp.sh
+	@./scripts/beszel-setup-smtp.sh
+
+caddy-monitoring: require-ansible ## stats.levkin.ca + status.levkin.ca on Caddy VM
+	@echo "Ensuring monitoring public proxies on caddy..."
+	$(ANSIBLE_PLAYBOOK) playbooks/caddy-monitoring-sites.yml $(ANSIBLE_ARGS)
+
 vault-import-env: ## Merge .env secrets into Ansible vault (usage: make vault-import-env [ENV_FILE=.env])
 	@chmod +x scripts/vault-import-env.sh
 	@ENV_FILE="$(or $(ENV_FILE),.env)" scripts/vault-import-env.sh "$(or $(ENV_FILE),.env)"
--- a/docs/guides/ci-runners-and-control.md
+++ b/docs/guides/ci-runners-and-control.md
@ -0,0 +1,67 @@
+# CI runners and Ansible control hosts
+
+**LAN:** `10.0.10.0/24` · **Inventory group:** `[qa]` and `[ansible]`
+
+These hosts are **not** Proxmox guests on pve10; they live on pve201 or as standalone VMs. Use this doc with [host-list.md](host-list.md) and [unifi-static-dhcp.md](unifi-static-dhcp.md).
+
+---
+
+## Summary
+
+| Inventory | IP | User | Role | Proxmox | Notes |
+|-----------|-----|------|------|---------|-------|
+| **git-ci-01** | `10.0.10.223` | `ladmin` | Gitea Actions (`act_runner`) | pve201 VM **115** | 2 cores, 4 GB RAM, 64 GB disk |
+| **sonarqube-01** | `10.0.10.54` | `ladmin` | SonarQube analysis | pve201 (verify VMID) | QA static analysis |
+| **ansibleVM** | `10.0.10.157` | `master` | Ansible control / automation | pve201 (verify VMID) | `become` via sudo; vault secrets in group_vars |
+
+---
+
+## git-ci-01 — Gitea Actions runner
+
+- **Host vars:** `inventories/production/host_vars/git-ci-01.yml`
+- **Runner config:** `/etc/act_runner/config.yaml` on the guest
+- **Capacity:** 2 concurrent jobs (`git_ci_runner_capacity: 2`)
+- **Maintenance:** weekly docker prune via `maintenance_cron` role
+
+**When pve201 is tight:** consider a second runner LXC on pve10 after Nextcloud/Portainer retire (see plan-2 capacity table).
+
+```bash
+make ping HOST=git-ci-01
+ssh ladmin@10.0.10.223
+```
+
+---
+
+## sonarqube-01 — code quality
+
+- **Inventory:** `[qa]` group
+- **Login:** `ssh ladmin@10.0.10.54` (key only after hardening)
+
+```bash
+make ping HOST=sonarqube-01
+```
+
+Pin **MAC → `10.0.10.54`** in UniFi if DHCP drift is observed ([unifi-static-dhcp.md](unifi-static-dhcp.md)).
+
+---
+
+## ansibleVM — control node
+
+- **Host vars:** `inventories/production/host_vars/ansibleVM.yml`
+- **Secrets:** `vault_ansiblevm_become_password` in vault
+- **Purpose:** run playbooks from the LAN when not using your Mac
+
+```bash
+make ping HOST=ansibleVM
+ssh master@10.0.10.157
+```
+
+On your Mac, the repo at `~/Documents/code/ansible` with `~/.ansible-vault-pass` remains the primary control path (`make apply`, etc.).
+
+---
+
+## Related
+
+- [host-list.md](host-list.md) — Proxmox guest IPs/MACs
+- [security-remediation-plan.md](security-remediation-plan.md) — SSH keys on QA hosts
+- [levkin-selfhost-plan-2.md](levkin-selfhost-plan-2.md) — dev-apps / runner migration backlog
--- a/docs/guides/handoff-2026-05-24.md
+++ b/docs/guides/handoff-2026-05-24.md
@ -0,0 +1,95 @@
+# Handoff — homelab sprint (2026-05-24)
+
+**Branch:** `homelab-sprint-2026-05-23` (or current working branch)  
+**Repo:** `/Users/macilia/Documents/code/ansible`  
+**Master plan:** [levkin-selfhost-plan-2.md](levkin-selfhost-plan-2.md) · **Status:** [homelab-status-2026-05-22.md](homelab-status-2026-05-22.md)
+
+---
+
+## Done this session
+
+### Monitoring (LXC 218 @ `10.0.10.22`)
+
+| Item | State |
+|------|--------|
+| **Kuma** | 17 HTTP monitors, all email-linked; admin password in vault |
+| **Kuma status page** | https://status.levkin.ca — slug `homelab`, 17 monitors, domain in Kuma |
+| **Umami** | https://stats.levkin.ca → `:3000` (Caddy + DNS) |
+| **Beszel** | 16 agents **up** (hub, pve10/pve201, P1 apps, vaultwarden, hermes, static LXCs) |
+| **SMTP** | `alerts@levkine.ca` — Kuma + Beszel; `listmonk@` — user UI + vault |
+
+### Caddy (VM 106 @ `10.0.10.50`)
+
+- `stats.levkin.ca` → `10.0.10.22:3000` (was already live)
+- `status.levkin.ca` → `10.0.10.22:3001` (added 2026-05-24)
+- Playbook: `playbooks/caddy-monitoring-sites.yml` · `make caddy-monitoring`
+
+### Infra / migrations
+
+- Nextcloud VM 201 retired; listmonk → LXC 221; pve201 VM 113 destroyed
+- Vikunja OIDC verified (browser login as `ilia`)
+- Gitea deploy key levkin LXC 220 — added + tested
+- UniFi API key in vault; DHCP fixed IPs applied for homelab VMs
+
+### Vault keys added/updated
+
+- `vault_uptime_kuma_password`, `vault_unifi_url`, `vault_unifi_api_key`, `vault_unifi_site`
+- Export: `make vault-export-env`
+
+---
+
+## Hostname map (monitoring)
+
+| DNS | Service | Backend | Public? |
+|-----|---------|---------|---------|
+| `stats.levkin.ca` | Umami | `:3000` | Yes (tracker + admin reachable) |
+| `status.levkin.ca` | Kuma status page | `:3001` | Yes (status only via Kuma domain binding) |
+| *(none)* | Beszel | `:8090` | **LAN/Tailscale only** |
+| *(none)* | Kuma admin | `:3001` | LAN only — do not expose `/dashboard` |
+| *(none)* | Dockge | `:5001` | LAN only |
+
+**Hermes ≠ Mattermost:** VM **117** @ `.36` = Hermes agent; Mattermost = VM **107** @ `slack.levkin.ca` (`.237` in Caddy today).
+
+---
+
+## Still open (manual / P1)
+
+1. **Beszel UI** — enable alerts per system (bell icon: Status, CPU, RAM, disk)
+2. **Rotate secrets** pasted in chat (UniFi key, Kuma password, Beszel admin if still temp)
+3. **Cal OIDC** — deferred (no `CALCOM_LICENSE_KEY`)
+4. **NAS / Jellyfin / DebianDesktop** — deferred
+5. **Optional:** `metrics.levkin.ca` → Beszel behind Tailscale/LAN-only Caddy block
+
+---
+
+## Key commands
+
+```bash
+make vault-export-env
+make caddy-monitoring          # stats + status on Caddy
+./scripts/kuma-add-monitors.sh # needs KUMA_PASSWORD in .env
+./scripts/beszel-setup-smtp.sh # Beszel SMTP via alerts@
+ssh root@10.0.10.22            # monitoring LXC
+ssh root@10.0.10.50            # Caddy VM
+```
+
+## Key files touched
+
+- `playbooks/caddy-monitoring-sites.yml` (new)
+- `scripts/kuma-add-monitors.sh`, `scripts/beszel-setup-smtp.sh`
+- `docs/guides/monitoring-stack.md`, `homelab-status-2026-05-22.md`, `levkin-selfhost-plan-2.md`
+- `inventories/production/group_vars/all/vault.yml` (encrypted — do not commit plaintext)
+
+## Beszel agent install quirk
+
+Installer hangs on “Enable automatic daily updates?” — use `printf n | install.sh`. SSH sessions may exit 255 after success; verify with `systemctl is-active beszel-agent`.
+
+## UniFi
+
+- API: `https://192.168.2.1/proxy/network/api` + header `X-API-KEY`
+- Network for homelab: **Administration** `10.0.10.0/24`
+- n8n MAC in live UniFi: `bc:24:11:c9:f7:48` @ `.154` (inventory `61:de:7a` is a different guest @ `.35`)
+
+---
+
+*Last updated: 2026-05-24*
--- a/docs/guides/homelab-status-2026-05-22.md
+++ b/docs/guides/homelab-status-2026-05-22.md
@ -1,6 +1,6 @@
-# Homelab status — 2026-05-23
+# Homelab status — 2026-05-24

-Quick checklist. **Master plan:** [levkin-selfhost-plan-2.md](levkin-selfhost-plan-2.md) · **Cursor plan:** `~/.cursor/plans/levkin_selfhost_rollout_e75909ae.plan.md`
+Quick checklist. **Master plan:** [levkin-selfhost-plan-2.md](levkin-selfhost-plan-2.md)

 ## Done (automation)

@ -8,42 +8,72 @@ Quick checklist. **Master plan:** [levkin-selfhost-plan-2.md](levkin-selfhost-pl
 |------|--------|
 | Mailcow `alerts@levkine.ca` | Created via API |
 | Kuma + Dockge + Umami | LXC 218 @ `10.0.10.22`; Dockge stack **monitoring** active |
+| **Kuma monitors** | **17** HTTP monitors; all linked to email alerts (`alerts@`) — [monitoring-stack.md](monitoring-stack.md#uptime-kuma--monitors) |
+| **Kuma admin password** | In vault (`vault_uptime_kuma_password`); `make vault-export-env` → `KUMA_PASSWORD` |
+| **Beszel hub + agents** | **16 systems all up** — hypervisors, P1 apps, vaultwarden, hermes, static LXCs — [monitoring-stack.md](monitoring-stack.md#4-beszel-agents-deployed) |
 | Old Kuma pve201 LXC 305 | Stopped, `onboot` off |
 | `stats.levkin.ca` | Caddy → Umami `:3000` |
 | Tracking scripts | levkin.ca + caseware + auto + portfolio (`iliadobkin.com`) |
 | **levkin.ca** | LXC **220** @ `10.0.10.60`; Caddy → nginx; `/` = spec, `/folders/` = stack |
 | Portfolio `iliadobkin.com` | Migrated pve201 LXC **306** → pve10 LXC **219** @ `10.0.10.106`; Caddy → nginx `:80` |
 | Kuma SMTP | Working (user confirmed) |
+| **DNS `levkin.ca` + `www`** | A → `142.180.237.136` verified 2026-05-23 |
+| **Nextcloud VM 201** | Stopped, `onboot 0`, Caddy block removed (~8 GiB RAM on pve10) |
+| **Listmonk** | pve201 VM **113** → pve10 LXC **221** @ `10.0.10.148`; SMTP `listmonk@` ✅ (user UI); pve201 VM **113 destroyed** |
+| **Phase 0 snapshots** | `pre-homelab-sprint-20260523` on pve10 217/218/210/106/201; pve201 106/113 + LXC 301 |
+| **Vikunja OIDC** | ✅ browser login as `ilia` verified 2026-05-24 — [vikunja-authentik-oidc.md](vikunja-authentik-oidc.md) |
 | Git remote | `git@git.levkin.ca:ilia/...` (SSH → `10.0.10.169` via `~/.ssh/config` on site LXCs) |
 | auto repo | Pushed/pulled on `git.levkin.ca` |
 | caseware repo | Pushed to Gitea via bundle on server; LXCs pull via internal SSH |
 | Vault | Mailcow, Umami, Mattermost in vault; `make vault-export-env` → `.env`; `make vault-pull-infra-secrets` = hosts → vault |
 | Caddy root SSH | Works (`make bootstrap-root-ssh-caddy`) |
-| Hermes Mattermost | `mattermost.env` on VM; Telegram optional/off |
+| Hermes agent (VM **117**) | @ `10.0.10.36` — Telegram bridge; **not** Mattermost |
+| Mattermost (VM **107**) | @ `slack.levkin.ca` — Kuma monitors this; separate from Hermes |
+| **CI / control docs** | [ci-runners-and-control.md](ci-runners-and-control.md) — git-ci-01, sonarqube-01, ansibleVM |
+| **SMTP inventory** | [smtp-inventory.md](smtp-inventory.md) — who uses which mailbox |
+
+## Capacity snapshot (2026-05-24)
+
+| Node | RAM available | Disk (local-lvm) | Notes |
+|------|---------------|------------------|--------|
+| **pve10** | ~22 GiB / 62 GiB | ~1.30 TiB free (~22% used) | Primary for new LXCs |
+| **pve201** | ~19 GiB / 125 GiB | ~922 GiB free (~46% used) | Do not add services |

 ## Your list — still to do

 ### You (UI / hardware / DNS)

 - [x] **Kuma SMTP** — working
- [ ] **DNS `levkin.ca` + `www`** — A records → home IP (`142.180.237.136`); apex currently parked at AWS, not homelab
- [ ] **Gitea deploy key (levkin LXC 220)** — add `deploy-levkin-levkin.ca` pubkey in repo settings (SSH pull); HTTPS clone works meanwhile
- [ ] **UniFi DHCP reservations** — [unifi-static-dhcp.md](unifi-static-dhcp.md) @ https://192.168.2.1/
- [ ] **Cal.com → Authentik OIDC** — **deferred** (no license key) — [cal-authentik-oidc.md](cal-authentik-oidc.md); Phase 4 → Vikunja — [sso-selfhosted-matrix.md](sso-selfhosted-matrix.md)
+- [x] **DNS `levkin.ca` + `www`** — A records → `142.180.237.136`
+- [x] **Gitea deploy key (levkin LXC 220)** — added + tested 2026-05-24
+- [x] **UniFi DHCP** — API reservations applied (gitea, vaultwarden, n8n, hermes, actual, jellyfin, mailcow, truenas, PVE.BU.SVR); key in vault
+- [x] **Beszel agents** — 16 systems (P1 + vaultwarden + hermes + statics) — [monitoring-stack.md](monitoring-stack.md#4-beszel-agents-deployed)
+- [x] **Kuma monitors** — levkin.ca, portfolio, search, pdf, umami script, mattermost added
+- [x] **Listmonk SMTP** — `listmonk@levkine.ca` updated in UI; vault synced
+- [ ] **Cal.com → Authentik OIDC** — **deferred** (no license key)
 - [x] **Portainer VM 109** — stopped and destroyed on pve10 (2026-05-23)
- [x] **Listmonk** — service was stopped; `listmonk.service` enabled on VM 113 (2026-05-23)
 - [x] **Mailcow** — LAN TCP timeout fixed (netfilter `MAILCOW` drop rule) — [mailcow-lan-proxy-fix.md](mailcow-lan-proxy-fix.md)
 - [ ] **DebianDesktop VM 100** — RAM lowered to 24 GB in Proxmox; **reboot guest** to apply balloon
- [ ] **Nextcloud VM 201 retire** — remove Kuma monitor, Caddy `nextcloud.levkin.ca`, stop VM
- [ ] **NAS.SP00 disk replace** — then start Jellyfin (VM 101)
+- [x] **Nextcloud VM 201 retire** — Caddy removed, VM stopped, `onboot 0`
+- [ ] **NAS.SP00 disk replace** — then start Jellyfin (VM 101) — **deferred (skip for P0)**
 - [x] **Gitea deploy key (portfolio)** — `git pull` works on LXC 219; Gitea VM SSH fixed (`/home/git/.ssh/authorized_keys` + `sudo` to `gitea`)
- [ ] **`.env`** — optional mirror: `make vault-export-env` (vault already has secrets)
+- [x] **pve201 VM 113** — destroyed 2026-05-24 after listmonk LXC verified
 - [ ] **Rotate** any secrets pasted in chat (Hermes token, etc.)

+### Next (manual / UI)
+
+1. **Beszel UI** — enable alerts per system (bell icon): Status (1 min), CPU/RAM/disk thresholds
+2. ~~**Vikunja**~~ — ✅ OIDC login as `ilia` verified
+3. ~~**UniFi DHCP**~~ — ✅ VM reservations via API (listmonk LXC 221 static via `pct set` — skip)
+4. ~~**Gitea deploy key**~~ — ✅ levkin LXC 220 tested
+5. ~~**Kuma status page**~~ — ✅ `status.levkin.ca` + Caddy
+
+**Deferred:** NAS SP00 / Jellyfin; DebianDesktop reboot; Cal OIDC (no license)
+
 ### Later / defer

 - [ ] Caddy → edge LXC `.20`
- [ ] Immich, Crater, Beszel
+- [ ] Immich, Crater
 - [ ] Public SSH for `git.levkin.ca:22` (optional Caddy `layer4` or DNS split)

 ## Site LXCs (marketing)
--- a/docs/guides/host-list.md
+++ b/docs/guides/host-list.md
@ -43,7 +43,8 @@ Update this file whenever a guest is created, migrated, or re-IP’d. See [levki
 | 219 | portfolio | **marketing site** | `10.0.10.106/24` | `10.0.10.106/24` | ✅ **Static** | `BC:24:11:DF:94:32` | Static HTML `/var/www/portfolio` → `iliadobkin.com` (migrated from pve201 LXC 306) |
 | 220 | levkin | **marketing site** | `10.0.10.60/24` | `10.0.10.60/24` | ✅ **Static** | `BC:24:11:C6:B2:E4` | Vite `www/` → `levkin.ca` (spec), `levkin.ca/folders` (stack) — [site-lxc-git.md](site-lxc-git.md) |
 | 217 | identity | identity | `10.0.10.21/24` | `10.0.10.21/24` | ✅ **Static** | `BC:24:11:3C:85:45` | Authentik + Postgres + Redis; `auth.levkin.ca` via Caddy |
-| 218 | monitoring | monitoring | `10.0.10.22/24` | `10.0.10.22/24` | ✅ **Static** | `BC:24:11:54:43:13` | Uptime Kuma `:3001`, Dockge `:5001`, Umami `:3000` — see [monitoring-stack.md](monitoring-stack.md) |
+| 218 | monitoring | monitoring | `10.0.10.22/24` | `10.0.10.22/24` | ✅ **Static** | `BC:24:11:54:43:13` | Kuma `:3001`, Dockge `:5001`, Umami `:3000`, Beszel `:8090` (LAN) — [monitoring-stack.md](monitoring-stack.md) |
+| 221 | listmonk | productivity | `10.0.10.148/24` | `10.0.10.148/24` | ✅ **Static** | `BC:24:11:18:0C:62` | Migrated from pve201 VM **113** 2026-05-23; Postgres 17 + native binary |

 **pve201 (not pve10):** LXC **305** `kuma-debian` @ `10.0.10.197` — **stopped 2026-05-22** (replaced by monitoring LXC 218). `onboot` disabled. LXC **306** `portfolio` — **destroyed/purged 2026-05-22** (now pve10 LXC **219** @ `10.0.10.106`).

@ -66,7 +67,7 @@ Update this file whenever a guest is created, migrated, or re-IP’d. See [levki
 | 150 | pihole00-debian | — | link-local* | TBD | ⏳ | `BC:24:11:86:76:97` | Running |
 | 117 | hermes | services | `10.0.10.36/24` | `10.0.10.36/24` | ⏳ stable DHCP | `BC:24:11:51:1E:99` | On pve10; guest agent; inventory `hermes` |
 | 200 | PVE.BU.SVR | labs | `10.0.10.200/24` | `10.0.10.200/24` | ⏳ stable DHCP | `BC:24:11:DA:95:3B` | Running |
-| 201 | NextcloudAIO-debian | (decommission) | `10.0.10.24/24` | — | 🗑️ **Retiring** | `BC:24:11:14:D4:DE` | Export done; remove Caddy + Kuma monitor, then stop VM |
+| 201 | NextcloudAIO-debian | (decommission) | `10.0.10.24/24` | — | 🗑️ **Stopped** | `BC:24:11:14:D4:DE` | Retired 2026-05-23 — Caddy removed, `onboot 0`, ~8 GiB RAM freed |
 | 300 | pihole-debian | — | — | — | — | — | **Stopped** |

 \* ARP showed IPv6 link-local only at audit time — confirm IPv4 inside guest or install QEMU guest agent.
@ -88,11 +89,11 @@ Update this file whenever a guest is created, migrated, or re-IP’d. See [levki
 | vaultwardenVM | `10.0.10.142` | VM 104 | ✅ |
 | giteaVM | `10.0.10.169` | VM 102 | ✅ |
 | n8n | `10.0.10.154` | VM 103? | ⚠️ verify (WRA vs n8n) |
-| listmonk | `10.0.10.148` | — | On **pve201** (`[comms]`) |
+| listmonk | `10.0.10.148` | LXC **221** | ✅ migrated from pve201 VM 113 |
 | mailcow | `10.0.10.132` | pve201 VM 106 | ✅ `[comms]` |
 | hermes | `10.0.10.36` | VM 117 | ✅ on pve10 |
 | jellyfin | `10.0.10.232` | VM 101 | ✅ (stopped until NAS healthy) |
-| nextcloud | `10.0.10.24` | VM 201 | commented out (retiring) |
+| nextcloud | `10.0.10.24` | VM 201 | stopped / retired (commented in inventory) |
 | portainerVM | — | VM 109 | removed (Dockge on monitoring) |

 ---
@ -126,6 +127,18 @@ pct set 216 -net0 name=eth0,bridge=vmbr0,ip=10.0.10.59/24,gw=10.0.10.1

 ---

+## QA / control (not pve10 LXCs)
+
+See [ci-runners-and-control.md](ci-runners-and-control.md).
+
+| Inventory | IP | Proxmox | Notes |
+|-----------|-----|---------|-------|
+| git-ci-01 | `10.0.10.223` | pve201 VM 115 | Gitea Actions runner |
+| sonarqube-01 | `10.0.10.54` | pve201 | SonarQube |
+| ansibleVM | `10.0.10.157` | pve201 | Ansible control (`master`) |
+
+---
+
 ## Audit checklist

 - [x] `pct list` / `qm list` on pve10
@ -138,5 +151,7 @@ pct set 216 -net0 name=eth0,bridge=vmbr0,ip=10.0.10.59/24,gw=10.0.10.1
 - [x] Caddy VM 106 static `.50`
 - [x] LXC backups `backup-20260522` on 217, 218
 - [ ] Router DHCP reservations for VMs — [vm-static-ip-router-reservations.md](vm-static-ip-router-reservations.md) (manual in router UI; table ready)
- [ ] Retire VM 201 (Nextcloud)
+- [x] Retire VM 201 (Nextcloud) — stopped 2026-05-23
+- [x] Listmonk → pve10 LXC 221 @ `.148`
+- [ ] UniFi: update listmonk DHCP MAC `BC:24:11:18:0C:62` (was VM 113 `…:11:53:9A`)
 - [ ] Re-run after NAS disk replace
--- a/docs/guides/levkin-selfhost-plan-2.md
+++ b/docs/guides/levkin-selfhost-plan-2.md
@ -17,20 +17,20 @@ Reference doc for the Proxmox homelab. Lives alongside the Cursor project that h
 |------|--------|
 | **Phase 0** Foundation | ✅ Mostly done — pve10 LXCs static; site LXCs 215/216/219/220 static; Caddy still on **VM 106** @ `.50` |
 | **Phase 1** Identity (Authentik) | ✅ LXC **217** @ `10.0.10.21` — admin + TOTP |
-| **Phase 2** Monitoring | ✅ LXC **218** @ `10.0.10.22` — Kuma, Dockge, Umami, Kuma SMTP |
+| **Phase 2** Monitoring | ✅ LXC **218** — Kuma (17 monitors), Dockge, Umami, Beszel (16 agents), SMTP |
 | **Phase 3** Cal.com | ✅ LXC **210** — booking + auto consult button; **OIDC deferred** (no enterprise license) |
 | **Phase 4** SSO | ⏳ **Next:** Vikunja → Authentik — [sso-selfhosted-matrix.md](sso-selfhosted-matrix.md) |
 | **Phase 5–8** | ⏳ Immich, Crater, Outline, automation depth — after P0 backlog |
 | **Comms health** | ✅ Mailcow + Listmonk restored 2026-05-23 — [mailcow-lan-proxy-fix.md](mailcow-lan-proxy-fix.md) |
 | **Site consolidation** | ⏳ **Partial** — git LXCs + levkin.ca LXC 220; optional later: static on Caddy VM |
 | **dev-apps** | ⏳ punimTag **9101** on pve201 until testing done |
-| **Nextcloud retire** | ⏳ VM **201** still running — **#1 RAM win on pve10** (~8 GiB) |
+| **Nextcloud retire** | ✅ VM **201** stopped, `onboot 0`, Caddy removed (~8 GiB RAM freed) |
 | **Portainer retire** | ✅ VM **109** destroyed 2026-05-23 (~16 GiB on pve10) |
 | **Security pass** | 🟡 Partial — SSH keys + apt + cron 2026-05-23 — [security-remediation-plan.md](security-remediation-plan.md) |

 ---

-## Capacity headroom (live check 2026-05-23)
+## Capacity headroom (live check 2026-05-24)

 Use this before adding LXCs/VMs. Re-check with `pvesm status` and `free -h` on each node.

@ -38,28 +38,30 @@ Use this before adding LXCs/VMs. Re-check with `pvesm status` and `free -h` on e

 | Resource | Total | Used | **Available** | Notes |
 |----------|-------|------|---------------|--------|
-| **local-lvm** (thin) | ~1.67 TiB | ~22% | **~1.30 TiB** | Plenty of disk for new LXCs |
-| **RAM** (host) | 62 GiB | ~28 GiB | **~33 GiB** | Portainer **109** removed 2026-05-23 |
+| **local-lvm** (thin) | ~1.67 TiB | ~22% | **~1.30 TiB** | New guests on **local-lvm** only (NAS SP00 degraded) |
+| **RAM** (host) | 62 GiB | ~40 GiB | **~22 GiB** | Portainer **109** + Nextcloud **201** freed |

-**Realistic new capacity on pve10 now:** ~**30+ GiB** headroom for Immich, Crater, Beszel, or **dev-apps** (6–8 GiB) after Nextcloud retires.
+**Running:** LXCs 210, 215–221; VMs 102–108, 117, 150, 200. **Stopped:** 101 Jellyfin, 201 Nextcloud.
+
+**Headroom:** ~**20+ GiB RAM** for Immich, Crater, or dev-apps LXC.

 **Still available to free:**

 | Stop / retire | Frees (maxmem) |
 |---------------|----------------|
 | ~~Portainer VM **109**~~ | ✅ **16 GiB** freed |
-| Nextcloud VM **201** | **8 GiB** ← do next |
+| ~~Nextcloud VM **201**~~ | ✅ **8 GiB** freed |
 | Hermes VM **117** (if not needed) | **16 GiB** |
 | Site LXCs 215/216 → Caddy static (optional) | **~1 GiB** |

-### pve201 (pve) — **do not add new services**
+### pve201 (pve) — **do not add new homelab services**

 | Resource | Total | Used | **Available** | Notes |
 |----------|-------|------|---------------|--------|
 | **local-lvm** | ~1.67 TiB | ~46% | **~922 GiB** | Disk OK |
-| **RAM** | 125 GiB | ~114 GiB | **~10 GiB** | GPU VM **104** (64 GB), DebianDesktop **100** (24 GB set — **reboot guest**), punimTag **9101** (16 GB) |
+| **RAM** | 125 GiB | ~105 GiB | **~19 GiB** | GPU **104** (64 GB), DebianDesktop **100** (reboot for 24 GB), punim **9101** (16 GB) |

-**Verdict:** New stacks belong on **pve10**. pve201 only benefits from **stopping/migrating** guests (punim after testing, GPU resize, old Kuma already stopped).
+**Verdict:** New stacks on **pve10** only. pve201: stop/migrate punim after testing.

 ---

@ -70,7 +72,7 @@ Use this before adding LXCs/VMs. Re-check with `pvesm status` and `free -h` on e
 - Mailcow — VM, mail domain is `levkine.ca` (with e)
 - Vaultwarden, Vikunja, n8n, Listmonk, Mattermost, Nextcloud — across various LXCs
 - **Cal.com** — LXC id `210`, `cal.levkin.ca`, Postgres included, admin user `ilia`, 15-min consult event live at `cal.levkin.ca/ilia/consult` with Jitsi link
- Caddy entries live for: `levkin.ca`, `caseware.levkin.ca`, `auto.levkin.ca`, `iliadobkin.com`, `cal.levkin.ca`, `listmonk.levkin.ca`, `pdf.levkin.ca`, `search.levkin.ca`, `auth.levkin.ca`, `stats.levkin.ca`
+- Caddy entries live for: `levkin.ca`, `caseware.levkin.ca`, `auto.levkin.ca`, `iliadobkin.com`, `cal.levkin.ca`, `listmonk.levkin.ca`, `pdf.levkin.ca`, `search.levkin.ca`, `auth.levkin.ca`, `stats.levkin.ca`, **`status.levkin.ca`**
 - **Authentik** — LXC **217** @ `10.0.10.21`, `https://auth.levkin.ca`, admin + TOTP enrolled
 - **Monitoring** — LXC **218** @ `10.0.10.22`: Uptime Kuma `:3001`, Dockge `:5001`, Umami `:3000` (LAN-only) — [monitoring-stack.md](monitoring-stack.md)
 - **Umami** + **Authentik** admin/TOTP/backup codes — done
@ -164,8 +166,9 @@ Reachable only via local network or Tailscale/Wireguard:

 | Subdomain | Service | Notes |
 |---|---|---|
-| `stats.levkin.ca` | Umami collector | Only the tracking script endpoint needs to be public; admin UI stays internal |
-| `status.levkin.ca` | Uptime Kuma | Kuma supports a separate public status page URL — that one can be public |
+| `stats.levkin.ca` | Umami | Public tracker script; admin UI prefer LAN `:3000` |
+| `status.levkin.ca` | Uptime Kuma | **Public status page** only (not admin UI) |
+| *(none)* | Beszel | **LAN/Tailscale** `10.0.10.22:8090` — host metrics, no public DNS |

 ---

@ -319,17 +322,29 @@ pct reboot <ID>

 ## Backlog (priority order)

-### P0 — next (ordered)
-1. ~~Umami / Kuma / Dockge~~ ✅
-2. ~~Portainer VM 109~~ ✅ (2026-05-23)
-3. **Retire Nextcloud VM 201** — ~8 GiB on pve10; remove Caddy + Kuma monitor
-4. **Vikunja → Authentik OIDC** — first real SSO ([sso-selfhosted-matrix.md](sso-selfhosted-matrix.md))
-5. **UniFi DHCP reservations** — [unifi-static-dhcp.md](unifi-static-dhcp.md)
-6. **DNS `levkin.ca` apex** → home IP (still parked at AWS)
-7. **Beszel** on monitoring LXC 218
-8. ~~Cal.com OIDC~~ — deferred until `CALCOM_LICENSE_KEY`; Authentik app `cal-com` ready
-9. **NAS.SP00** disk replace → Jellyfin VM 101
-10. **DebianDesktop VM 100** — reboot for 24 GB limit on pve201
+### P0 — status (2026-05-24)
+
+| # | Item | Status |
+|---|------|--------|
+| 1 | Umami / Kuma / Dockge | ✅ |
+| 2 | Portainer VM 109 | ✅ removed |
+| 3 | Nextcloud VM 201 | ✅ retired |
+| 4 | Listmonk → LXC 221 | ✅ + SMTP + VM 113 destroyed |
+| 5 | Beszel agents | ✅ **16 systems** |
+| 6 | Kuma monitors + email | ✅ **17 monitors**, all alert-linked |
+| 7 | DNS `levkin.ca` apex | ✅ |
+| 8 | Vikunja OIDC infra | ✅ live — browser test as `ilia` still manual |
+| 9 | UniFi DHCP listmonk MAC | ⏳ manual @ UniFi |
+| 10 | NAS / Jellyfin / DebianDesktop | **deferred** |
+| 11 | Cal OIDC | deferred (no license) |
+
+### P1 — next (manual / UI)
+
+1. **Beszel alerts** — bell icon per system in hub UI
+2. **Vikunja** — browser login as `ilia`
+3. **UniFi** — listmonk MAC `BC:24:11:18:0C:62`
+4. **Gitea deploy key** — levkin LXC 220
+5. **Kuma status page** — optional `status.levkin.ca`

 ### P1 — when ready
 - **Outline** — wiki for client docs
@ -381,11 +396,12 @@ See **[homelab-status-2026-05-22.md](homelab-status-2026-05-22.md)** for automat
 | 2 | **Cal.com → Authentik OIDC** | ⏸ **deferred** | — | Needs `CALCOM_LICENSE_KEY`; infra ready — [sso-selfhosted-matrix.md](sso-selfhosted-matrix.md) |
 | 3 | **auto.levkin.ca** → Cal booking link | ✅ | — | Consult button live |
 | 4 | **Stop Portainer VM 109** | ✅ | — | Removed 2026-05-23; **~16 GiB RAM** on pve10 |
-| 5 | **Retire Nextcloud VM 201** | ⏳ **next** | 30 min | **~8 GiB RAM** on pve10 |
-| 6 | **Vikunja → Authentik OIDC** | ⏳ | 1–2 h | Phase 4 kickoff |
+| 5 | **Retire Nextcloud VM 201** | ✅ | — | ~8 GiB RAM freed |
+| 6 | **Vikunja → Authentik OIDC** | 🟡 infra OK | 15 min | Browser login as `ilia` |
 | 7 | **UniFi DHCP reservations** | ⏳ | 20 min | [unifi-static-dhcp.md](unifi-static-dhcp.md) |
-| 8 | **DNS levkin.ca apex** | ⏳ | 15 min | AWS parked → `142.180.237.136` |
-| 9 | **Beszel** on 218 | ⏳ | 1 h | Capacity before Immich |
+| 8 | **DNS levkin.ca apex** | ✅ | — | `142.180.237.136` |
+| 9 | **Beszel + Kuma** | ✅ | — | 16 Beszel agents; 17 Kuma monitors |
+| 10 | ~~**Listmonk SMTP**~~ | ✅ | — | UI + vault |
 | 10 | **NAS.SP00** disk → Jellyfin | ⏳ hardware | — | VM 101 |
 | 11 | **DebianDesktop reboot** | ⏳ | 5 min | Apply 24 GB on pve201 |
 | 12 | **Caddy → edge LXC `.20`** | ⏳ defer | ~30 min | Phase 1.5 |
@ -398,8 +414,8 @@ See **[homelab-status-2026-05-22.md](homelab-status-2026-05-22.md)** for automat

 | Want to add… | Node | RAM budget | Prerequisite |
 |--------------|------|------------|--------------|
-| Small app (Mealie, Linkwarden) | pve10 | 2 GB LXC | Stop 109 and/or 201 first if host feels tight |
-| Medium (Outline, Crater) | pve10 | 4 GB LXC | Free **~24 GiB** via Portainer + Nextcloud retire |
+| Small app (Mealie, Linkwarden) | pve10 | 2 GB LXC | ~22 GiB free on pve10 |
+| Medium (Outline, Crater) | pve10 | 4 GB LXC | Portainer + Nextcloud already freed |
 | Heavy (Immich + ML) | pve10 or pve201 GPU | 4–8 GB+ | NAS healthy; pve201 only after GPU/punim sized down |
 | Dev sandbox | pve10 `dev-apps` | 6–8 GB | punim 9101 migration only after testing |

--- a/docs/guides/monitoring-stack.md
+++ b/docs/guides/monitoring-stack.md
@ -11,8 +11,126 @@ All admin UIs are **LAN-only** (no public Caddy blocks). Use Tailscale or local
 | **Uptime Kuma** | http://10.0.10.22:3001 | 3001 | Admin + monitors configured ✅ (replaces pve201 LXC **305** @ `.197`, stopped) |
 | **Dockge** | http://10.0.10.22:5001 | 5001 | Manage compose on **this LXC only** |
 | **Umami** | http://10.0.10.22:3000 | 3000 | Password changed ✅; levkin.ca + caseware + auto + portfolio tracked |
+| **Beszel** | http://10.0.10.22:8090 | 8090 | Hub (LAN). **Fresh install 2026-05-23** — create admin on first visit; SMTP below |

-Secrets: `/opt/monitoring/.env` on the LXC (mode 600). Not in git.
+| Service | Public URL | Admin / internal | Role |
+|---------|------------|------------------|------|
+| **Umami** | `stats.levkin.ca` (script + optional admin) | `http://10.0.10.22:3000` | Page analytics |
+| **Uptime Kuma** | **`status.levkin.ca`** (status page only — configure in Kuma) | `http://10.0.10.22:3001` | URL uptime + alerts |
+| **Beszel** | **No public hostname** (by design) | `http://10.0.10.22:8090` | Host metrics (CPU/RAM/disk) |
+| **Dockge** | — | `http://10.0.10.22:5001` | Compose UI on monitoring LXC |
+
+Optional later: `metrics.levkin.ca` → Beszel **only if** you add Caddy with Tailscale/LAN restriction — not a public dashboard like `status.levkin.ca`.
+
+---
+
+## Beszel — admin + SMTP (Mailcow)
+
+**URL:** http://10.0.10.22:8090
+
+### 1. Create admin
+
+Open in browser → **Create account** (first user = admin).
+
+### 2. SMTP via `alerts@levkine.ca`
+
+Same Mailcow mailbox as [Kuma email alerts](#uptime-kuma--email-alerts-mailcow).
+
+**UI:** Settings → Notifications → SMTP (PocketBase mail settings)
+
+| Field | Value |
+|-------|--------|
+| Host | `mail.levkine.ca` |
+| Port | `587` |
+| Username | `alerts@levkine.ca` |
+| Password | alerts mailbox password |
+| Enforce TLS | **OFF** (STARTTLS, like Kuma) |
+| Sender | `alerts@levkine.ca` |
+
+After save: `ssh root@10.0.10.22 'cd /opt/monitoring && docker compose restart beszel'`
+
+**Script (after admin exists):**
+
+```bash
+export BESZEL_EMAIL='you@example.com' BESZEL_PASSWORD='...' SMTP_PASS='...'
+./scripts/beszel-setup-smtp.sh
+```
+
+### 3. Local agent (monitoring LXC)
+
+Hub → **Add System** → host `/beszel_socket/beszel.sock` → copy **TOKEN** and hub **KEY**.
+
+In `/opt/monitoring/compose.yml`, set `TOKEN` and `KEY` **literally** in the `beszel-agent` environment block (do not rely on `${BESZEL_*}` — compose may expand them empty):
+
+```yaml
+  beszel-agent:
+    environment:
+      LISTEN: /beszel_socket/beszel.sock
+      HUB_URL: http://127.0.0.1:8090
+      TOKEN: "<system-token-from-hub>"
+      KEY: "<hub-public-key-from-settings>"
+```
+
+Then: `cd /opt/monitoring && docker compose up -d beszel-agent`
+
+**Hypervisor agents (pve10, pve201):** install script flags are `-k`, `-t`, `-url`, `-p` (not `-key`):
+
+```bash
+curl -fsSL https://get.beszel.dev | sh -s -- \
+  -k 'ssh-ed25519 AAAA...' \
+  -t '<system-token>' \
+  -url 'http://10.0.10.22:8090' \
+  -p 45876
+```
+
+Verify: `systemctl status beszel-agent` and hub shows **up**.
+
+**Deployed 2026-05-24:** **16 systems, all up** — see table below.
+
+### 4. Beszel agents (deployed)
+
+| System | Host | Status |
+|--------|------|--------|
+| monitoring-218 | unix socket | ✅ |
+| pve10 / pve201 | hypervisors | ✅ |
+| identity-217 | `10.0.10.21` | ✅ |
+| caddy-106 | `10.0.10.50` | ✅ |
+| mailcow | `10.0.10.132` | ✅ |
+| listmonk-221 | `10.0.10.148` | ✅ |
+| cal-210 | `10.0.10.228` | ✅ |
+| gitea-102 | `10.0.10.169` (Alpine/openrc) | ✅ |
+| vikunja-301 | `10.0.10.159` | ✅ |
+| vaultwarden-104 | `10.0.10.142` (`ladmin` + sudo) | ✅ |
+| hermes-117 | `10.0.10.36` (Hermes agent VM; **not** Mattermost — `ladmin` + sudo) | ✅ |
+| levkin-220 | `10.0.10.60` | ✅ |
+| caseware-215 | `10.0.10.105` | ✅ |
+| auto-216 | `10.0.10.59` | ✅ |
+| portfolio-219 | `10.0.10.106` | ✅ |
+
+**Install notes:** use `-k`, `-t`, `-url`, `-p` flags; pipe `printf n` to skip auto-update prompt. Gitea needs `apk add curl` first. Vaultwarden/hermes use `ladmin` + sudo.
+
+#### Still optional (Beszel)
+
+| System | Notes |
+|--------|--------|
+| **n8n** VM 103 | Automation |
+| **TrueNAS** VM 105 | When NAS healthy |
+
+#### Alerts to enable (UI — bell icon per system)
+
+| Alert | Suggested threshold |
+|-------|---------------------|
+| **Status** | Down > 1 min |
+| **CPU** | > 85% for 5 min |
+| **Memory** | > 90% |
+| **Disk** | > 85% on `/` or `local-lvm` |
+| **Temperature** | > 75°C (if sensors available) |
+
+**Optional later:** universal token (`/settings/tokens`) for bulk agent install — requires a regular Beszel user (superusers cannot use universal tokens).
+
+**Not Beszel:** use **Kuma** for public URL uptime (`https://…`); use **Umami** for page analytics (no alerts).
+
+See also [smtp-inventory.md](smtp-inventory.md) — Beszel shares `alerts@` with Kuma; Umami does not use SMTP.

 ---

@ -38,15 +156,46 @@ Optional off-node copy (when NAS healthy): `vzdump 217 218 --storage local --mod

 ## Uptime Kuma — monitors

-Configured in UI (all green). **Remove** the Nextcloud monitor when VM 201 is retired.
+**URL:** http://10.0.10.22:3001

-| Name | URL |
-|------|-----|
-| Authentik | https://auth.levkin.ca |
-| Cal.com | https://cal.levkin.ca |
-| Caseware / Auto | marketing sites |
-| Mailcow | https://mail.levkine.ca |
-| Listmonk, Gitea, Vault, Todo, PVE nodes | per your dashboard |
+### Current (2026-05-24) — 17 monitors
+
+| Name | URL | Email alert |
+|------|-----|-------------|
+| Authentic | https://auth.levkin.ca | ✅ |
+| Cal.com | https://cal.levkin.ca | ✅ |
+| Caseware Landing | https://caseware.levkin.ca | ✅ |
+| Automation Landing | https://auto.levkin.ca | ✅ |
+| Mailcow | https://mail.levkine.ca | ✅ |
+| Listmonk | https://listmonk.levkin.ca | ✅ |
+| Gitea | https://git.levkin.ca | ✅ |
+| Todo (Vikunja) | https://todo.levkin.ca | ✅ |
+| Vault | https://vault.levkin.ca | ✅ |
+| PVENAS | https://10.0.10.10:8006 | ✅ |
+| PVE201 | https://10.0.10.201:8006 | ✅ |
+| **levkin.ca** | https://levkin.ca | ✅ added 2026-05-24 |
+| **Portfolio** | https://iliadobkin.com | ✅ |
+| **Search** | https://search.levkin.ca | ✅ |
+| **PDF** | https://pdf.levkin.ca | ✅ |
+| **Umami script** | https://stats.levkin.ca/script.js | ✅ |
+| **Mattermost** | https://slack.levkin.ca | ✅ (VM **107** — not Hermes) |
+
+SMTP notification **My Email Alert** → `alerts@levkine.ca` → `idobkin@gmail.com`. All monitors linked to email alerts.
+
+**Public status page:** https://status.levkin.ca — slug `homelab`, domain set in Kuma (17 monitors). Caddy → `10.0.10.22:3001`. Admin UI stays at `http://10.0.10.22:3001` (LAN only).
+
+Admin password in vault: `vault_uptime_kuma_password` (`admin` user).
+
+### Add monitors (script)
+
+```bash
+make vault-export-env   # sets KUMA_PASSWORD
+./scripts/kuma-add-monitors.sh
+```
+
+**Removed / do not add:** Nextcloud (`cloud.levkin.ca`) — VM 201 retired.
+
+**Not in Kuma (by design):** Beszel `:8090`, Dockge `:5001`, Umami admin `:3000` — LAN-only; use **Beszel** for host metrics.

 ---

--- a/docs/guides/smtp-inventory.md
+++ b/docs/guides/smtp-inventory.md
@ -0,0 +1,53 @@
+# SMTP / mail senders — homelab inventory
+
+**Mail server:** Mailcow @ `mail.levkine.ca` (domain **`levkine.ca`** with **e**)
+
+**Alerts mailbox (`alerts@levkine.ca`):** homelab monitoring only — Kuma + Beszel. Password in vault: `vault_alerts_mailbox_password`, `vault_kuma_smtp_password`, `vault_mailcow_mailbox_passwords.alerts`.
+
+---
+
+## Who sends email?
+
+| Mailbox / sender | Service | Host | Purpose | SMTP configured? |
+|------------------|---------|------|---------|------------------|
+| **`alerts@levkine.ca`** | **Uptime Kuma** | LXC 218 | Monitor down/up emails → `idobkin@gmail.com` | ✅ live (vault) |
+| **`alerts@levkine.ca`** | **Beszel** | LXC 218 | Resource / status alerts | ✅ live (vault) |
+| **`cal@levkine.ca`** | **Cal.com** | LXC 210 | Booking confirmations, calendar mail | ✅ `EMAIL_SERVER_*` in `/opt/cal/.env` |
+| **`listmonk@levkine.ca`** | **Listmonk** | LXC 221 | Newsletter campaigns, bounces | ✅ live (UI + vault) |
+| **Umami** | LXC 218 | Analytics only | Page views via `stats.levkin.ca` | **No SMTP** |
+| **Authentik** | LXC 217 | OIDC / optional email flows | Optional — not required for Vikunja SSO | ⏳ optional |
+| **Mattermost** | VM **107** @ `slack.levkin.ca` | Team chat | Separate from **Hermes** agent VM **117** @ `.36` | ⏳ optional |
+| **Mailcow** | pve201 VM 106 | Inbound + relay | Server itself — not an app client | N/A |
+| **pote / MirrorMatch** | app LXCs (when deployed) | App-specific mail | Separate vault keys | ⏳ per app |
+
+---
+
+## Umami and alerts
+
+**Umami does not need SMTP.** It collects page views via `stats.levkin.ca/script.js`. You log into the dashboard at `http://10.0.10.22:3000` with `UMAMI_ADMIN_PASSWORD` (vault). There is nothing to “alert” from Umami — use **Kuma** (uptime) and **Beszel** (CPU/RAM/disk) for operational alerts.
+
+---
+
+## Scripts
+
+| Script | Mailbox |
+|--------|---------|
+| `scripts/kuma-setup-smtp.sh` | `alerts@` |
+| `scripts/beszel-setup-smtp.sh` | `alerts@` |
+| `scripts/mailcow-mailbox.sh` | any Mailcow mailbox |
+
+```bash
+make vault-export-env
+./scripts/beszel-setup-smtp.sh   # BESZEL_EMAIL, BESZEL_PASSWORD, SMTP_PASS
+```
+
+---
+
+## Action items
+
+1. ~~**Listmonk**~~ ✅ — SMTP in UI; vault `vault_listmonk_smtp_password` + Mailcow synced
+2. ~~**Beszel agents**~~ ✅ — monitoring LXC + pve10 + pve201 ([monitoring-stack.md](monitoring-stack.md#3-local-agent-monitoring-lxc))
+3. ~~**Beszel agents**~~ ✅ — 16 systems
+4. ~~**Kuma monitors**~~ ✅ — 17 monitors + email alerts
+5. **Beszel alerts** — in hub UI, enable Status/CPU/RAM/disk alerts per system
+6. **Cal.com** — uses its own mailbox; rotate via Mailcow if needed (not `alerts@`).
--- a/docs/guides/unifi-static-dhcp.md
+++ b/docs/guides/unifi-static-dhcp.md
@ -73,7 +73,7 @@ This guide is for **VMs** (and pve201 guests) that still use DHCP.
 | 4 | hermes | 10.0.10.36 | BC:24:11:51:1E:99 | |
 | 5 | actual | 10.0.10.158 | BC:24:11:10:7B:64 | |
 | 6 | jellyfin | 10.0.10.232 | BC:24:11:29:B8:84 | stopped until NAS OK |
-| 7 | listmonk (pve201 VM 113) | 10.0.10.148 | BC:24:11:11:53:9A | |
+| 7 | ~~listmonk LXC 221~~ | 10.0.10.148 | BC:24:11:18:0C:62 | **Skip** — static via `pct set`, not DHCP |
 | 8 | Mailcow (pve201) | 10.0.10.132 | BC:24:11:34:75:2D | |
 | 9 | TrueNAS | 10.0.10.107 | BC:24:11:14:DE:B5 | optional pin |
 | 10 | PVE.BU.SVR | 10.0.10.200 | BC:24:11:DA:95:3B | lab VM |
--- a/docs/guides/vm-static-ip-router-reservations.md
+++ b/docs/guides/vm-static-ip-router-reservations.md
@ -5,7 +5,9 @@ Proxmox **LXCs** use `pct set … ip=10.0.10.X/24` (done for 210, 215–219).
 **VMs** without cloud-init are pinned by **router DHCP reservation by MAC** (Method B in plan-2).
 Ansible **cannot log into your router** — configure static leases in the UI.

-**Your UniFi:** https://192.168.2.1/ — step-by-step: [unifi-static-dhcp.md](unifi-static-dhcp.md).
+**UniFi API:** key in vault (`vault_unifi_api_key`). Reservations can be applied via API (see homelab status).
+
+---

 Homelab guests use **`10.0.10.0/24`** (gateway `10.0.10.1`). If UniFi also serves `192.168.2.x`, ensure the `10.0.10.x` segment is the network those VMs/LXCs use
 (or that routing/DHCP relay matches your Proxmox bridge).
@ -30,7 +32,7 @@ Homelab guests use **`10.0.10.0/24`** (gateway `10.0.10.1`). If UniFi also serve
 | 200 | PVE.BU.SVR | `BC:24:11:DA:95:3B` | `10.0.10.200` | — |
 | 201 | NextcloudAIO | `BC:24:11:14:D4:DE` | `10.0.10.24` | **decommission** — skip new work |
 | 101 | Jellyfin | `BC:24:11:29:B8:84` | `10.0.10.232` | stopped |
-| 113 | listmonk (pve201) | `BC:24:11:11:53:9A` | `10.0.10.148` | listmonk |
+| 113 | listmonk (pve201) | `BC:24:11:11:53:9A` | `10.0.10.148` | **retired** → pve10 LXC **221** MAC `BC:24:11:18:0C:62` |
 | — | Mailcow (pve201 VM 106) | `BC:24:11:34:75:2D` | `10.0.10.132` | mailcow (inventory) |

 After reserving in the router, mark **DHCP/Static** as `✅ router` in [host-list.md](host-list.md).
--- a/inventories/production/group_vars/all/vault.example.yml
+++ b/inventories/production/group_vars/all/vault.example.yml
@ -44,9 +44,16 @@ vault_kuma_smtp_user: "alerts@levkine.ca"
 vault_kuma_smtp_password: "CHANGE_ME"
 vault_kuma_smtp_to: "idobkin@gmail.com"

+# UniFi Network (Integrations → API key)
+vault_unifi_url: "https://192.168.2.1"
+vault_unifi_site: "default"
+vault_unifi_api_key: "CHANGE_ME"
+
 # Umami (monitoring LXC /opt/monitoring/.env)
 vault_umami_db_password: "CHANGE_ME"
 vault_umami_app_secret: "CHANGE_ME"
+# Umami admin UI password (monitoring LXC :3000)
+vault_umami_admin_password: "CHANGE_ME"

 # Cal.com ↔ Authentik OIDC (make cal-oidc)
 vault_cal_oidc_client_secret: "CHANGE_ME"
--- a/inventories/production/group_vars/all/vault.yml
+++ b/inventories/production/group_vars/all/vault.yml
@ -1,142 +1,170 @@
 $ANSIBLE_VAULT;1.1;AES256
-62616334383737633962313839313235653935663832623061333532616566343565626437376230
-3333393831623434663736656331303462626534626265380a356135653866666438373838663137
-61373962356364306365323933386262613837333364356564383163383638363430323230393430
-3032346238343264340a636539663735396335313135363330373536353562666537653764643637
-36663437366166616437303738646466656331313266653431303462366532616639323136346137
-66663932346561333535303438623734643864613330396331626161616265393731633365393930
-37326565363931386532623432343339656534393032663634353961306330303737313765333330
-65316436383030666564663537323937666634343966653562353434333537366338393838333666
-30356339353732623932393665663237343630303533363232336263323732376461353338663831
-62666365333330353361373732306436623637623932636235393434323339663266396631346237
-63393762643338346563643637666135336139336461333537373137626464613339373937383830
-39643039363234346134663062373130343230663839613234373838393434373532313732656332
-63373739616163666361666330393866396331616136383565383763303563323261323330313832
-64386661383838366336633335323431356133366162373464313533653734613366623537646636
-65323862376466343530303439396639616135373030613638363630313264623337653233636532
-38383664613337303565336136333434613638663239393234656534353264623166333837376436
-36633837613339613161363764383538303363323232346636313862393930343333633131383833
-65316166363062363330373734323232366136653030336439343932613337623662383236663834
-66303137353438373661633537633333633733666663393435383436396634393739383039383139
-32653438303134326663653164633039653435643766616637313433623463366531633962613434
-33396262333739643865346465363862303337356239663337356330363232383331346435393930
-64306633363064656566346662363433313434376631663032343635656463313530626635623930
-65383434663064666535613561313265616436326533313336303836386635343134626361343566
-36653233656337613838323164376666656338383337633065393237373737623934626265343133
-38393763316132373234623735353731656261643736353562616361643033303064393962343239
-32623363653466363565323436643639643934663530646333356532363463363564363862373232
-38396535393034653565643236363733393032306335363934623462386639363961306265646636
-32383738653633613732313030626135353366626537646263303634323539343866363033643337
-66396235323461666131643030353164616265623635636438363738653233363435353761366531
-66623033656331386138623864363461333933653636653566303733616137303030663430643535
-64386534346463393638613764353966343837333235623262343164326564616138353731363663
-39666634323663373831326664326337656164323738326335663734373538303135653861393362
-65303865366235333538623330373032306661386436323530336631616639366636376135303537
-31373634636561356239366437623637363735653633316634353862666139303565393533643864
-64656335356236353232303135616265666266376634313437633236666461343233333732323832
-61313230393162383163336634303066613664376338633964643431346335616533396466393736
-31383862666365633665623766643665623361376565386531323234303236393162356331346535
-65353231316531326438343237633133393361336366353232623866393138376232643133326161
-63333236626237613536323964356435383933646264656137623632343665393530343463343230
-61343464383230366339616439343762626435303832393462666463363030383365343938666264
-33636437333266656130633365666162316366616262386436333861373533343433356633356630
-31643666626262386535626233653337303861666666653366643361363164353430643561613532
-32373239373038306533393464373365323638653630306630363931623931336663666339356464
-64646634356437326435656163306562346530363435336138353330356162333431353466313763
-64666538666332653762633064653664663531373638393530653034323864383938346631303165
-62343163636366633161383464626639633638323363306139626632343836646135346332393235
-62376536316164636631626639656533323337366335616534356538386266343436613530653131
-36383733373637303864636334633237663331623663663562613261393736323137373130613537
-61353431396139663861366639616631613064323230366131373666373964393738623936393431
-62366530623938373836636265393233663661663664613430366237396637366561616433333463
-64623335303834376432383361396433373537633066333937663633663433333339343262363338
-38633532366334313164346236646665663363623065666331613961653639313563316563383231
-64343834373066316233336465366634306537303666383831306237396362366663343430353162
-35643638653234396134653638653663333765313236313764383835343431303134383537313237
-35626563376163643336623534633236313363383062373437666536306462383632626332643430
-39326661633134393465363333356136323361363831363961646230393561663838313935386432
-66653430613231343731623630313362366138613465373631653632303139636438656439633361
-31326262313431363536633434346431626336623139333235363338626435666439616433323931
-30386238663931393066353237616537366434363536306163613931306138653364623663666438
-63396331313438623662393532333834376237343462313263626139366133353131313164613861
-38313632386336646362313634633938383963306339383362633236653235353061626337353936
-32626464343166323438616637346661633861396264633365386638666538333932633530306139
-64393132613562353835323162346532313262353266366230393839323462626362353533323834
-35393261373039336537623339613463613335363362353438623837376631646233653362383636
-31613261323361623934653939613661333836666637383534643137346261353333303861363665
-62386237646661626536363034313833363965373562316334336232643164633436333261303730
-39386233646162323365393034663137636462316432333335313366363933633065323264646136
-64326338303766613230393539626430326263646631646536623436643734376237373031316466
-62306136373465633130653564633233356331313761366333623363646666313365623563346334
-31666535346461336630626466616664363330626338333961386239333663326536316266346634
-33323064333161313232343239346439623633346161346465313532383061376137323839666365
-64376132306338663565623531623136663436333730366563313261626661373233393438646561
-61396562666533316635363432306139366430333837333866346436306135333862663734306164
-38313766666230393861323632316231343362656136366338336564623431373662333366323833
-33613232326230643530356137646635623030316663343466666666333734636230346263353365
-65363637356635666638613566613131383864316465666536336333393334653436666261393461
-64376639303632626165613361346636303064333532613064303032643562396262623632396539
-61623333643630616536393163623330353164383864623064383732633733353630323534663732
-36383133633066393263363533616334653933336235333938663132366334326234386264386531
-62373466393234666563613637313136663764376239666434383038613932376532653531613164
-34613834313532383165336634613536636437393638653964393831393533303630333933636464
-31613634346235396331386534303636313066396361393138393635633134353035613863656364
-63363030376662356636333566373063613433373330383139396530316163303633656438326333
-65353435613561303539326538613261393339616537373136313030656133323766396464646634
-34363061366166303465616133663835323232373763336634386231396230383965306164393731
-39633333353936666361656530363665383039626533333035373663326537373538633864626366
-64613435663834666137326335333736376466356236353637333262373834336131393733646138
-31363630626432643061663538626230356637373863643866326530373962393065393464663466
-37326165653235653166386561363339353331663164326639616135663736316363336531333439
-35363033343934323063613133326264313665613363386464303662633333646330373637636366
-37396562303164636261323633373538323835623235396161303964373735356538393431303031
-64663636326364386266306434343361353439616533303632363165376639313635663637623263
-37376233323233663364393439663137396265646230613631383039316230356539316130353062
-33303732323063633738666636623737366631336164396637396533656364316333616536336632
-34303963623031353137626331623031326136373538633336633835623337303831616365323066
-35333931393136393965623135626363393335306363366639323034633064663035613566313037
-38616234666131343064633561326466326365643863653664623932333734643332383963356665
-64326435643333333435636665383165386364663134613564386639346566353831343239646239
-35376338386631646236303031303665336166643437316131626438646237663331306438666130
-32323539393431303039393964363161633461303136616430666539393162633464623436656638
-61363736363665633965656362643432376266393531333539633737343165313562616133366131
-34346266323931363137303463666363336163373839306533393831323262313861393333643336
-37626239366432393461613630366636366631353237396461663566333935343037336438626262
-33306264613065373638373634303262626338386236386533616563633131366665663738353837
-33333936306266633965613338393662656161613465396163653438306463303138656536366531
-66343634306332313561386531373663343535343232646162396361626666633034663133613364
-65323536346264636164616463626535353261396362633736376531666334346537666562363339
-39653430386565313731346230653632613830396165313561333865333234656532383339313065
-30663565393030343134383536336335616537333336396232333839373533353161623264626434
-61393334316331613739666434653839353933336332396536313937323939646264313133373863
-66643138656661336264646338376232396138616465373562393063333336343036326632306662
-66303836326636663264633334356533613066383935316635313236633631376633613535303830
-33646566346661346539633638363135343939653363623232313864613132393235643961633566
-34653464303430313466326463346563363964363666623665633265356138336133616261333839
-37343036363065613766366565343765306663623037383933323230646566333935306564343039
-30343730633135643338366262376365326561353538346433636336633866393565326334326431
-64613136353139316331343333643564343534643931313164323934373465386437376637613838
-65333237386462666262326663316639393961363033656233356330666634366633373336326531
-33303535323036663837363537366436653930353637353962393464373361323166663031343532
-37383735613334316434356232343466373539666562326430656538653634323361363236313030
-34313537633433666333356661383838663861613765383564633835333437363330616163316432
-30323762656230323035663139323363346235633337346637663632383762393363396632613631
-35303161383263613164303535633063346432643563363436306665613738346338666336646530
-36383639353032636133353438396362333763623164376338653564616465303538646432353763
-66663262636661363465326463326639613431643065623966373630323161356565326362646635
-32633335303339633232396166393235643462356565323236356539653033363663333262386235
-36316432386165366530323737353862393263343063343138343334343966313838336639646463
-36303137323961626561343238323634373830323161303365306465373036323262663835376630
-34376662356238643939613536383432393464656530326530333262356162623531636364363662
-65613166383563333237376135656362306362366434346565366235626532623964303661626632
-61646462633533663830613436633937336364643562653362616464636130343264666233333932
-37323736316539636336633163643166333231376464656462666364303761313962366635663336
-31353738396532616137333033313362393830663434323236313031623863643735323838646561
-34633065623764333734353166323234633538363230633865353764333663613239306664386232
-65306661333939336634343535393261326335663163663431633630373936336465623634376362
-31393231313435306564333234633938353336366239646637366162343065366261303538613962
-36323065663362383538633536393161653332383035336236363364373133326366366130626135
-34346237366338663962643966613363336165633765663137653930323731393235616137613364
-37623462396333376263326364363166613831396161393933623532346637326262616434636265
-64323336626663303131323331376330393232666233626662363264616533646462323233333633
-3535
+65613063646137663934353263656638653161613265383666626435656466646465396236386162
+3237303634613436386234663239313939373238326663630a663963616465373836333837343935
+61356266646463313730383831656165663062623739303132313865336433626138363430663033
+3266663236663730340a383335383938363930353635646365373536643335313564623238313032
+61383637613635643434306465623230343864663039363633346431306162643364383661616366
+36376666343066316338343731643163626435333134356461343065366635666130383132363934
+37656565346436643666363439323661333561613934303766646632666438623331663666333738
+64626464343033366162333531646261623439346639626266646563633565626232333133313938
+35633836626639376565363639666235643633343233663539613861363461313433333936616531
+30356532323565373462373533663962343463613466373731643263393863396433313935363036
+61663263303266313665656133366135616564363137623166306638613434613566666364613132
+34613066626366313363383834376661313764663737383162343663333361313666386332376562
+31616438333438346537366664333130646264636461346362666335343038643765353264396338
+35633530613237646235353436303236313561363134356566356466376539393033663564343334
+36336130313262343537333934383532386639386664346561346131663437343564623736643863
+30336334396235326164333439656336613935653937636233643136653062643663643466366436
+32336533336566626266613838343364626439376563663462366664356235326535643732303531
+35643739373939626663633861343065346262623765373339643732323666396236626130646537
+34336530633337633662306363656661303464623961333335353538623462656162323634326264
+39386130323935303437346263646633356233326339663534363037326365666630356434356133
+61316363306462383661666232383131353132646432656531626463666534626561616537653735
+31663531333834363062633436626631383536633631343237373034633131326633353333303663
+64336536653663323132386564306431336436323635346430613661353165376365363063633735
+62313033353161373232653031306362376330343535306533663839373535643165613636313532
+65393932623666613431313431373761313666313462633937646633656436363661353537313463
+30313365346636326236366261316135663962663065323531643938343464376163353034313563
+64326438343937326361303137373036373364643235346230646333633662643738363833383036
+31336537376664343530333638613665376666393432643434373361646166623032333437376164
+38336130393465353839383330393931336461326230306136323265653664643831366363633635
+63343136336466356533633633653931616236656432636434343130653733306161623538666262
+33653933663034376434353966633761666162353166616636376561306266653930386639613935
+30343162396133633136373038663965633536363731613832636534303766663031333135386238
+66333334356263383661363737373363636535363238363761656363373164613034386136396332
+63663836623532653136623663396336613066303839643534363365363031653335396365643362
+32663364616438656565656533313733653933656538616137336531643864393766333935626333
+64343835303739303461333430653430386364336262656163313837363634366335666163366232
+34323338386234393431643865666334333065393534306438386335626438393130333162623337
+38393561353339336265613236366631376561356538303836663465346532306166363466393865
+36343339373032643536386364343332663632313763646139303962646266663266633465343463
+61303737353761656363663432643133643738623965353139336434646530323339356163303065
+36623230396535386639313534393239343439613238656361616333656139316465323163653639
+66653630666139393730313136326532636439666631313363656464663563326634613134356432
+39613866633165306132643761646136646335663831656337343235643365376130636666316632
+64656236333064353332396264316134633565323332396136356636353430313064396338613266
+65656338303266333735386537616337643838663136663237373862363163623238303139303661
+66636232346339323338653264643939643630616238623363636336666362343864663730306633
+33326566363961633066373732393439346262323561643534353063633939633731336331623332
+63623562313863383966656437643033656664623132613965613435643065336133643661386434
+34366161343038306337393761613661353163623061376362653061383739376338323463623161
+65393531643061653264323837313236333733393536653733656336666464353635653434383738
+33656363643264383161383032333365633766316335636334346662623335396163613635363365
+34383066376433663336633030616362366633613366633832333034303136656264393464366538
+33366266613336373739376230393364653862666261356664323464643033336162386337653937
+32666363376463383461343864343265393062636237633438643238386436616163323463663732
+66613165333762633961663833376465643537323062613664306135663332323038353934383530
+31383235393034376532386265346332303634346631653537303836383234386239343732386363
+39363232346130633864343364636434646439663834326635666666376664366430303932613232
+66643832363836336430383763383635336136313639313665366330353231333265663164323037
+65636535633531393439386331383037326361366232613332653637373435333233643138306333
+38343137386131613937346630353332306639616335646665613166326634636339346262356538
+33386534366465633532323837663464306133343762623862363536643139353461663132313337
+64336231363537623435663430623336326537386362326165306139343165353136346336643437
+38626636613666336161666439343862363738366531343832303237393734653937336637636534
+62313731303162653863306635373538333334303663663635363732353833323735353130393531
+65316236333133396634373165393365653837653066343839333334366362636537653466383439
+66363339366265353834663033653135333562373364653830663730386130353732393766653931
+64333334363634356135666664633164366665383234623563333330306335303836333662316633
+35343864363938383461336436366566323734306264393763396265663961313534323663613631
+35346439666639613836613162303736323564653761363661383265356538663633393838613635
+65323939663164613432643136363233336436633131356661313034376266643832386137653565
+65356235616165643265346661396434626539336633353437316339626139383362343834646332
+62393632313130323866336233396362303437653032653662613332616430666139376561663164
+65393633356364313937326565616164373463653736336533616565616635366663336533363936
+34636336626364363039383231363962363065316436336234346463323035303665383437383938
+36366537393864363365393633363937373162323237306432636266386462346464393661643032
+31616632383964646462316666393664366232633736356166323432333834323261393564656530
+66316433333037313930646234666162336331333861643162633434306432653061613830613861
+31363266303635613830373265383436363833653237386436653036666336326137386232646535
+33353930373038376332343766393338383235653338316564646632643138373033663433613832
+66336562333532643330343962323063383138633737633634386661623539353431313730313666
+65633337613632646635643566393136353430366539613862323932633631656662376564623562
+31623666303637393862353233653533616633643537626431373066616338326662313939343236
+33633534646633633161306336663139623036303330613364373539626433333937633631346534
+31636632393561333961623565663637663237636262636366336437326634306331313938653561
+61353263643263383266333932373933353834303663316337303763643336376165656135323331
+34613131373638303730383961383738613164386339646231663533393931343236333761376630
+33393133313134323135393533653064626635363532323866623033336437653732336237616232
+38353034306362656234663030313435306234353139376135383233363264613135303937663637
+31353530333162613563626435316231303037353639653366353032393937306134373935356564
+37386364333533626638383435303365356465336637383933386463623930356633326331316231
+64353334353764383764326131326236353532666330303765336265356632356433303338323831
+32336334373433623134373765393365613465316462646439623366626631353338613464643139
+33313634626663623961653632633064393364336434373137306261316264663061313065396536
+32326634643164623635666335363839356362633038636330626663326130636363616635333732
+61306135643830613433343131623263626265373161646266666234383838376566356365653036
+35303235353537333831623066356462633135316464663133633933356565323538373732313839
+39636236343136363233613734366539656139636666303266323162316235343030633662343636
+39353465333034303838663264323532663832373932306638323539353938313338373966346339
+33653563386136623239303561616235623439616236336239353730326264353261633433393638
+64656465356662303266656235303361363462376430386137363330633339643961343332393266
+37636364623766616265616430636666636130353765363636656361356632356436626232336663
+31373336663136383738336436373539373962613161363738393761666636396330386238646566
+31633831646366383439363431376335333464363234353632633062653432376339353863616539
+64666463316662626233643964363237326638613335623766323535636263313033363236323963
+65383935643035313062383135653361666331616232393262343562643435666362373564323263
+38333039663831343236333231313964323734623532663336323636363230343530633432346662
+31353931636336346462643365626635336639666339393331366665333161353566623963616236
+38363430316339326536366165616564336630626662383737346433636539303463643863373863
+66636231336238396265623139336630613933383166383734333837616639373536333265656237
+63666262383265396261316564336635373262313164383061646364356131396331323135323266
+33383030643732343938383637663465316463346432643064653035303865383433653333353437
+63633864353235656662346462666165383432333062353433623738376233346639663234376461
+35306137333963393537616635326536386339653734316138363165656431653232323963366635
+63373263643133393966396163333932613233373637326639373635363837656437353533623862
+66356336353163303638316364313339333533653265356138383763326633633761396238326336
+64643966333832633431613532623738666535393361323631636335653133363636663562376535
+64323539306434633165356134623566666564323437316535666235383162646163343761343331
+62353036663033333932333238646464656261336363303639303163303636383639613265396632
+64653137313330643133316664303764336536666534336238643833663165613231396366393739
+32323133616334373062306530343462353563616663333536373665323532636561303435386362
+38636539623036326432323465303833316436373135326335313061363130383139343933346661
+31326638356262613866613131376438396662616434643638326561613365623134656535373335
+62363435626431393632636338353632656430613564336166353339636232316234666430326431
+38653966303736303964653737323032336533313332643534303334613261646530396630633833
+35646137323766373536613339663333643134326335306635303834366330333132613230616439
+31353066303733313338636561613931303931663263326530353138323863643466663261333965
+39313363383030333130376331616533643437393830316661383766613834646663653435323364
+30316536336236336262643434636538326137373139363832313962663638333863346432633863
+37623039373763653339653235363866666639343039323636633864373034323761636365666263
+61313235613764626335313666366663383663663532353738636635653861643161623736306635
+65353030646331313565653362356463323265653962306236326263323561316661626463636633
+38366133363034623630346235313237376332666537373762363861373465376534646233636230
+34353636316464346662663530613437643436373931623966383566633739653834366662633538
+30343766366430636333646463393238633963343932313536366432336266613166323939613533
+62353533346130346235336262653561366436376362633966336439663838356431303039626335
+61626331333737623133356432666137393463666661386664373630306666616337666432363031
+33616330376630653538323636663930393033313965343830636566343234653361333135313766
+64666365343862343333626666656333323431623061656534343765643338333434393061326538
+36353331363131386334356136383439613731333766353833643364363665343937653636306464
+36373562333164633637373162633430353266343465613635326536663962633335303063376663
+61353235353034356638376532323530323138383465346234643639636136636532356666336236
+61633936623937366131343836633261643238643663353136313963303161313936393838323963
+63383361653834366136333936373463333634326635316132323632613533343933613434643031
+34333962353831643933303362313033643736353163306464373231353633373261663361613065
+34653165643636383935343162653034333438306437616263366130393664666661616465343339
+39303065643037636639633334656430656334643038306561393330366365613731363336333638
+35316137633262343833636631373063316361316535373536653561363463326437353464343161
+31386237363766346434353032306561643436393935303836363339316134303330306239306166
+64366161646664373535666333303763306431316639343262663762303239303339323063303462
+38373235656137303832623865376131616365376538613766373331633831346665323962316364
+33396337363235613162376162373733373366613462373432353330363762393062353436316539
+65643839333461366564336461346565333638643836653063373464376365326433613939666463
+37616235306632373262396435343264313338386364613132363537623762653530383537653236
+31303466623362356366663731376465363661613434623834663432633133656132376131376332
+62653766326137396665666330336632306263663464383263313732326338363937363566393733
+61633862376162323536363366396434313732366262646535333831663032333137373639383066
+35363235356430643965343665306566633630623863343866613530626662623063616434623037
+38383938656465643364636333316461613833623464366635396365393766383163633439396236
+62623930316339306235646431383866663038626135356337343262346562336166306532653833
+36613136376136633463636261646566323564363862613163646433643266643231343337653865
+38303866326164663030666635323738346536376462326631343439393364623636396665343937
+63346566663832313536646435626437383335623232366234626665323663356431643264353630
+32396130633337333164346662343539383432623265653066373665343532653435653835656561
+63383431663563636437626435333632303765653264303435633862323466336533333765666161
+34313931386236313338633265323330653233393835663035353061346533353736316633353666
+30343934363834386234306332383737373938363236623163376565643235626165303738353563
+35636264663937316532653934313730393934633833653130383461343763623636633836646632
+61623135623561636430366661373565616463643765306562383963343736306135353434336166
+3466
--- a/inventories/production/host_vars/listmonk.yml
+++ b/inventories/production/host_vars/listmonk.yml
@ -1,3 +1,5 @@
 ---
-# listmonk VM on pve201 — plain vars; secrets in vault
-# Previous fully-encrypted host_vars file moved to listmonk.yml.vault-bak (broken for Ansible merge).
+# listmonk LXC 221 on pve10 @ 10.0.10.148 — plain vars; secrets in vault
+# Migrated from pve201 VM 113 on 2026-05-23.
+proxmox_vmid: 221
+proxmox_node: PVENAS
--- a/inventories/production/hosts
+++ b/inventories/production/hosts
@ -36,7 +36,7 @@ ansibleVM ansible_host=10.0.10.157 ansible_user=master
 [comms]
 # pve201 — email + newsletters
 mailcow ansible_host=10.0.10.132 ansible_user=root url=https://mail.levkine.ca proxmox_vmid=106 proxmox_node=pve201
-listmonk ansible_host=10.0.10.148 ansible_user=root url=https://listmonk.levkin.ca proxmox_node=pve201
+listmonk ansible_host=10.0.10.148 ansible_user=root url=https://listmonk.levkin.ca proxmox_vmid=221 proxmox_node=PVENAS

 [services]
 # VMID 117: on PVENAS (pve10)
--- a/playbooks/caddy-monitoring-sites.yml
+++ b/playbooks/caddy-monitoring-sites.yml
@ -0,0 +1,72 @@
+---
+# Playbook: caddy-monitoring-sites
+# Purpose: stats.levkin.ca (Umami) + status.levkin.ca (Kuma status page) on Caddy VM 106
+# Targets: caddy
+# Usage: make caddy-monitoring
+
+- name: Ensure monitoring public proxies on Caddy
+  hosts: caddy
+  become: true
+  become_method: ansible.builtin.su
+
+  tasks:
+    - name: Ensure stats.levkin.ca block exists
+      ansible.builtin.shell: |
+        set -euo pipefail
+        if grep -q '^stats\.levkin\.ca {' /etc/caddy/Caddyfile; then
+          exit 0
+        fi
+        awk -v upstream="{{ monitoring_umami_upstream | default('10.0.10.22:3000') }}" '
+        /^caseware\.levkin\.ca \{/ { in_cw=1 }
+        in_cw && /^}$/ && !done {
+          print
+          print ""
+          print "stats.levkin.ca {"
+          print "    import security-headers"
+          print "    encode gzip"
+          print "    reverse_proxy " upstream
+          print "}"
+          done=1
+          next
+        }
+        { print }
+        ' /etc/caddy/Caddyfile > /tmp/Caddyfile.new
+        mv /tmp/Caddyfile.new /etc/caddy/Caddyfile
+      args:
+        executable: /bin/bash
+      register: stats_block
+      changed_when: stats_block.rc == 0
+      notify: Reload caddy
+
+    - name: Ensure status.levkin.ca block exists
+      ansible.builtin.shell: |
+        set -euo pipefail
+        if grep -q '^status\.levkin\.ca {' /etc/caddy/Caddyfile; then
+          exit 0
+        fi
+        awk -v upstream="{{ monitoring_kuma_upstream | default('10.0.10.22:3001') }}" '
+        /^stats\.levkin\.ca \{/ { in_stats=1 }
+        in_stats && /^}$/ && !done {
+          print
+          print ""
+          print "status.levkin.ca {"
+          print "    import security-headers"
+          print "    encode gzip"
+          print "    reverse_proxy " upstream
+          print "}"
+          done=1
+          next
+        }
+        { print }
+        ' /etc/caddy/Caddyfile > /tmp/Caddyfile.new
+        mv /tmp/Caddyfile.new /etc/caddy/Caddyfile
+      args:
+        executable: /bin/bash
+      register: status_block
+      changed_when: status_block.rc == 0
+      notify: Reload caddy
+
+  handlers:
+    - name: Reload caddy
+      ansible.builtin.command: caddy reload --config /etc/caddy/Caddyfile
+      changed_when: true
--- a/scripts/beszel-setup-smtp.sh
+++ b/scripts/beszel-setup-smtp.sh
@ -0,0 +1,137 @@
+#!/usr/bin/env bash
+# Configure Beszel (PocketBase) SMTP for Mailcow alerts mailbox.
+#
+# Prerequisite: admin account created at http://10.0.10.22:8090
+#
+# Usage:
+#   export BESZEL_URL=http://10.0.10.22:8090
+#   export BESZEL_EMAIL=you@example.com
+#   export BESZEL_PASSWORD='your-beszel-password'
+#   export SMTP_PASS='alerts@ mailbox password'   # or source .env
+#   ./scripts/beszel-setup-smtp.sh
+#
+# Optional: SMTP_TO for test email (default idobkin@gmail.com)
+
+set -euo pipefail
+
+REPO_ROOT="$(cd "$(dirname "${BASH_SOURCE[0]}")/.." && pwd)"
+BESZEL_URL="${BESZEL_URL:-http://10.0.10.22:8090}"
+BESZEL_EMAIL="${BESZEL_EMAIL:-}"
+BESZEL_PASSWORD="${BESZEL_PASSWORD:-}"
+SMTP_HOST="${SMTP_HOST:-mail.levkine.ca}"
+SMTP_PORT="${SMTP_PORT:-587}"
+SMTP_USER="${SMTP_USER:-alerts@levkine.ca}"
+SMTP_PASS="${SMTP_PASS:-}"
+SMTP_TO="${SMTP_TO:-idobkin@gmail.com}"
+
+if [[ -f "${REPO_ROOT}/.env" ]]; then
+  # shellcheck disable=SC1091
+  set -a
+  source "${REPO_ROOT}/.env"
+  set +a
+  SMTP_PASS="${SMTP_PASS:-${ALERTS_PASSWORD:-}}"
+fi
+
+if [[ -z "${BESZEL_EMAIL}" || -z "${BESZEL_PASSWORD}" ]]; then
+  echo "Set BESZEL_EMAIL and BESZEL_PASSWORD (Beszel admin you just created)" >&2
+  exit 1
+fi
+
+if [[ -z "${SMTP_PASS}" ]]; then
+  echo "Set SMTP_PASS or ALERTS_PASSWORD (alerts@levkine.ca mailbox password)" >&2
+  exit 1
+fi
+
+export BESZEL_URL BESZEL_EMAIL BESZEL_PASSWORD SMTP_HOST SMTP_PORT SMTP_USER SMTP_PASS SMTP_TO
+
+"${REPO_ROOT}/.venv/bin/python3" <<'PY'
+import json
+import os
+import sys
+import urllib.error
+import urllib.request
+
+base = os.environ["BESZEL_URL"].rstrip("/")
+email = os.environ["BESZEL_EMAIL"]
+password = os.environ["BESZEL_PASSWORD"]
+smtp_host = os.environ["SMTP_HOST"]
+smtp_port = int(os.environ["SMTP_PORT"])
+smtp_user = os.environ["SMTP_USER"]
+smtp_pass = os.environ["SMTP_PASS"]
+smtp_to = os.environ["SMTP_TO"]
+
+
+def req(method, path, token=None, body=None):
+    url = f"{base}{path}"
+    data = None
+    headers = {"Content-Type": "application/json"}
+    if body is not None:
+        data = json.dumps(body).encode()
+    if token:
+        headers["Authorization"] = token
+    request = urllib.request.Request(url, data=data, headers=headers, method=method)
+    try:
+        with urllib.request.urlopen(request, timeout=30) as resp:
+            raw = resp.read().decode()
+            return resp.status, json.loads(raw) if raw else {}
+    except urllib.error.HTTPError as e:
+        raw = e.read().decode()
+        print(f"HTTP {e.code} {path}: {raw}", file=sys.stderr)
+        raise
+
+
+print(f"Login to Beszel at {base} as {email}...")
+for collection in ("_superusers", "users"):
+    try:
+        status, auth = req(
+            "POST",
+            f"/api/collections/{collection}/auth-with-password",
+            body={"identity": email, "password": password},
+        )
+        token = auth.get("token")
+        if token:
+            print(f"Login OK ({collection})")
+            break
+    except urllib.error.HTTPError:
+        token = None
+        continue
+else:
+    print("Login failed: check BESZEL_EMAIL and BESZEL_PASSWORD", file=sys.stderr)
+    sys.exit(1)
+
+print("Configuring SMTP (Mailcow STARTTLS)...")
+req(
+    "PATCH",
+    "/api/settings",
+    token=token,
+    body={
+        "smtp": {
+            "enabled": True,
+            "host": smtp_host,
+            "port": smtp_port,
+            "username": smtp_user,
+            "password": smtp_pass,
+            "tls": False,
+            "authMethod": "PLAIN",
+            "localName": "monitoring.levkin.ca",
+        },
+        "meta": {
+            "senderName": "Beszel",
+            "senderAddress": smtp_user,
+        },
+    },
+)
+print("SMTP settings saved")
+
+print(f"Sending test email to {smtp_to}...")
+req(
+    "POST",
+    "/api/settings/test/email",
+    token=token,
+    body={"email": smtp_to, "template": "verification", "collection": "_superusers"},
+)
+print("Test email request accepted — check inbox (and restart beszel if alerts fail later)")
+
+print("\nDone. Restart hub to avoid cached mail client issues:")
+print("  ssh root@10.0.10.22 'cd /opt/monitoring && docker compose restart beszel'")
+PY
--- a/scripts/kuma-add-monitors.sh
+++ b/scripts/kuma-add-monitors.sh
@ -1,10 +1,11 @@
 #!/usr/bin/env bash
 # Add or update Uptime Kuma HTTP monitors via API.
 # Usage:
-#   source <(./scripts/vault-export-env.sh)   # or export KUMA_* manually
+#   export KUMA_PASSWORD='...'   # not in vault yet — set manually once
 #   ./scripts/kuma-add-monitors.sh
 #
 # Monitors are idempotent: existing names are skipped.
+# Links the default SMTP notification to any monitor that has none.

 set -euo pipefail

@ -13,18 +14,16 @@ KUMA_URL="${KUMA_URL:-http://10.0.10.22:3001}"
 KUMA_USER="${KUMA_USER:-admin}"
 KUMA_PASSWORD="${KUMA_PASSWORD:-}"

-if [[ -z "${KUMA_PASSWORD}" ]]; then
-  if [[ -f "${REPO_ROOT}/.env" ]]; then
-    # shellcheck disable=SC1091
-    set -a
-    source "${REPO_ROOT}/.env"
-    set +a
-    KUMA_PASSWORD="${KUMA_PASSWORD:-}"
-  fi
+if [[ -z "${KUMA_PASSWORD}" ]] && [[ -f "${REPO_ROOT}/.env" ]]; then
+  # shellcheck disable=SC1091
+  set -a
+  source "${REPO_ROOT}/.env"
+  set +a
+  KUMA_PASSWORD="${KUMA_PASSWORD:-}"
 fi

 if [[ -z "${KUMA_PASSWORD}" ]]; then
-  echo "Set KUMA_PASSWORD (or run vault-export-env.sh first)" >&2
+  echo "Set KUMA_PASSWORD (admin UI password; not stored in vault yet)" >&2
  exit 1
 fi

@ -40,16 +39,14 @@ except ImportError:
    print("Run: .venv/bin/pip install uptime-kuma-api", file=sys.stderr)
    sys.exit(1)

+# Public HTTPS endpoints worth watching (Beszel covers host metrics separately).
 MONITORS = [
-    {
-        "type": "http",
-        "name": "Gitea",
-        "url": "https://git.levkin.ca/user/login",
-        "interval": 60,
-        "retryInterval": 60,
-        "maxretries": 3,
-        "accepted_statuscodes": ["200-299"],
-    },
+    {"type": "http", "name": "levkin.ca", "url": "https://levkin.ca", "interval": 60, "retryInterval": 60, "maxretries": 3, "accepted_statuscodes": ["200-299"]},
+    {"type": "http", "name": "Portfolio", "url": "https://iliadobkin.com", "interval": 60, "retryInterval": 60, "maxretries": 3, "accepted_statuscodes": ["200-299"]},
+    {"type": "http", "name": "Search", "url": "https://search.levkin.ca", "interval": 120, "retryInterval": 60, "maxretries": 3, "accepted_statuscodes": ["200-299"]},
+    {"type": "http", "name": "PDF", "url": "https://pdf.levkin.ca", "interval": 120, "retryInterval": 60, "maxretries": 3, "accepted_statuscodes": ["200-299"]},
+    {"type": "http", "name": "Umami script", "url": "https://stats.levkin.ca/script.js", "interval": 300, "retryInterval": 120, "maxretries": 3, "accepted_statuscodes": ["200-299"]},
+    {"type": "http", "name": "Mattermost", "url": "https://slack.levkin.ca", "interval": 120, "retryInterval": 60, "maxretries": 3, "accepted_statuscodes": ["200-299"]},
 ]

 url = os.environ["KUMA_URL"]
@ -59,6 +56,12 @@ password = os.environ["KUMA_PASSWORD"]
 with UptimeKumaApi(url) as api:
    api.login(user, password)
    existing = {m.get("name"): m for m in api.get_monitors()}
+    notifs = [n for n in api.get_notifications() if n.get("isActive")]
+    smtp = next((n for n in notifs if n.get("isActive")), None)
+    if not smtp and notifs:
+        smtp = notifs[0]
+    if not smtp:
+        smtp = {"id": 1}  # fallback: first notification in DB

    for spec in MONITORS:
        name = spec["name"]
@ -67,4 +70,20 @@ with UptimeKumaApi(url) as api:
            continue
        result = api.add_monitor(**spec)
        print(f"added: {name} -> {result}")
+        existing[name] = result if isinstance(result, dict) else {"id": result}
+
+    if smtp:
+        nid = smtp["id"]
+        for m in api.get_monitors():
+            nlist = m.get("notificationIDList") or {}
+            if isinstance(nlist, dict) and nlist.get(str(nid)):
+                continue
+            if isinstance(nlist, dict):
+                nlist[str(nid)] = True
+            else:
+                nlist = {str(nid): True}
+            api.edit_monitor(m["id"], notificationIDList=nlist)
+            print(f"linked notification {nid} -> {m.get('name')}")
+    else:
+        print("warn: no SMTP notification found — create one in Kuma UI first", file=sys.stderr)
 PY
--- a/scripts/vault-export-env.sh
+++ b/scripts/vault-export-env.sh
@ -35,6 +35,8 @@ MAP = {
    "vault_mattermost_url": "MATTERMOST_URL",
    "vault_mattermost_token": "MATTERMOST_TOKEN",
    "vault_mattermost_allowed_users": "MATTERMOST_ALLOWED_USERS",
+    "vault_unifi_api_key": "UNIFI_API_KEY",
+    "vault_unifi_url": "UNIFI_URL",
 }

 def parse_env(text):
--- a/scripts/vault-import-env.sh
+++ b/scripts/vault-import-env.sh
@ -52,6 +52,8 @@ MAP = {
    "MATTERMOST_URL": "vault_mattermost_url",
    "MATTERMOST_TOKEN": "vault_mattermost_token",
    "MATTERMOST_ALLOWED_USERS": "vault_mattermost_allowed_users",
+    "UNIFI_API_KEY": "vault_unifi_api_key",
+    "UNIFI_URL": "vault_unifi_url",
    "PROXMOX_PASSWORD": "vault_proxmox_password",
    "LXC_ROOT_PASSWORD": "vault_lxc_root_password",
 }