ansible/docs/guides/monitoring-stack.md

Monitoring stack (LXC 218)

Host: monitoring @ 10.0.10.22 (PVENAS pve10, VMID 218)
Compose: /opt/monitoring/compose.yml
Stacks dir (Dockge): /opt/stacks

All admin UIs are LAN-only (no public Caddy blocks). Use Tailscale or local network.

Service	URL	Port	Notes
Uptime Kuma	http://10.0.10.22:3001	3001	Admin + monitors configured ✅ (replaces pve201 LXC 305 @ .197, stopped)
Dockge	http://10.0.10.22:5001	5001	Manage compose on this LXC only
Umami	http://10.0.10.22:3000	3000	Password changed ✅; levkin.ca + caseware + auto + portfolio tracked
Beszel	http://10.0.10.22:8090	8090	Hub (LAN). Fresh install 2026-05-23 — create admin on first visit; SMTP below

Service	Public URL	Admin / internal	Role
Umami	stats.levkin.ca (script + optional admin)	http://10.0.10.22:3000	Page analytics
Uptime Kuma	status.levkin.ca (status page only — configure in Kuma)	http://10.0.10.22:3001	URL uptime + alerts
Beszel	No public hostname (by design)	http://10.0.10.22:8090	Host metrics (CPU/RAM/disk)
Dockge	—	http://10.0.10.22:5001	Compose UI on monitoring LXC

Optional later: metrics.levkin.ca → Beszel only if you add Caddy with Tailscale/LAN restriction — not a public dashboard like status.levkin.ca.

---

Beszel — admin + SMTP (Mailcow)

URL: http://10.0.10.22:8090

1. Create admin

Open in browser → Create account (first user = admin).

2. SMTP via alerts@levkine.ca

Same Mailcow mailbox as Kuma email alerts.

UI: Settings → Notifications → SMTP (PocketBase mail settings)

Field	Value
Host	mail.levkine.ca
Port	587
Username	alerts@levkine.ca
Password	alerts mailbox password
Enforce TLS	OFF (STARTTLS, like Kuma)
Sender	alerts@levkine.ca

After save: ssh root@10.0.10.22 'cd /opt/monitoring && docker compose restart beszel'

Script (after admin exists):

export BESZEL_EMAIL='you@example.com' BESZEL_PASSWORD='...' SMTP_PASS='...'
./scripts/beszel-setup-smtp.sh
make beszel-setup-alerts   # Status + CPU/RAM/disk on all systems

3. Local agent (monitoring LXC)

Hub → Add System → host /beszel_socket/beszel.sock → copy TOKEN and hub KEY.

In /opt/monitoring/compose.yml, set TOKEN and KEY literally in the beszel-agent environment block (do not rely on ${BESZEL_*} — compose may expand them empty):

  beszel-agent:
    environment:
      LISTEN: /beszel_socket/beszel.sock
      HUB_URL: http://127.0.0.1:8090
      TOKEN: "<system-token-from-hub>"
      KEY: "<hub-public-key-from-settings>"

Then: cd /opt/monitoring && docker compose up -d beszel-agent

Hypervisor agents (pve10, pve201): install script flags are -k, -t, -url, -p (not -key):

curl -fsSL https://get.beszel.dev | sh -s -- \
  -k 'ssh-ed25519 AAAA...' \
  -t '<system-token>' \
  -url 'http://10.0.10.22:8090' \
  -p 45876

Verify: systemctl status beszel-agent and hub shows up.

Deployed 2026-05-24: 16 systems, all up — see table below.

4. Beszel agents (deployed)

System	Host	Status
monitoring-218	unix socket	✅
pve10 / pve201	hypervisors	✅
identity-217	10.0.10.21	✅
caddy-106	10.0.10.50	✅
mailcow	10.0.10.132	✅
listmonk-221	10.0.10.148	✅
cal-210	10.0.10.228	✅
gitea-102	10.0.10.169 (Alpine/openrc)	✅
vikunja-301	10.0.10.159	✅
vaultwarden-104	10.0.10.142 (ladmin + sudo)	✅
hermes-117	10.0.10.36 (Hermes agent VM; not Mattermost — ladmin + sudo)	✅
levkin-220	10.0.10.60	✅
caseware-215	10.0.10.105	✅
auto-216	10.0.10.59	✅
portfolio-219	10.0.10.106	✅

Install notes: use -k, -t, -url, -p flags; pipe printf n to skip auto-update prompt. Gitea needs apk add curl first. Vaultwarden/hermes use ladmin + sudo.

Still optional (Beszel)

System	Notes
n8n VM 103	Automation
TrueNAS VM 105	When NAS healthy

Alerts to enable (UI — bell icon per system)

Alert	Suggested threshold
Status	Down > 1 min
CPU	> 85% for 5 min
Memory	> 90%
Disk	> 85% on / or local-lvm
Temperature	> 75°C (if sensors available)

Optional later: universal token (/settings/tokens) for bulk agent install — requires a regular Beszel user (superusers cannot use universal tokens).

Not Beszel: use Kuma for public URL uptime (https://…); use Umami for page analytics (no alerts).

See also smtp-inventory.md — Beszel shares alerts@ with Kuma; Umami does not use SMTP.

---

Backups (pve10)

Guest	VMID	Snapshot	Date
identity	217	backup-20260522	2026-05-22
monitoring	218	backup-20260522	2026-05-22

On pve10:

pct listsnapshot 217
pct listsnapshot 218
# Rollback if needed:
# pct rollback 217 backup-20260522

Optional off-node copy (when NAS healthy): vzdump 217 218 --storage local --mode snapshot --compress zstd

---

Uptime Kuma — monitors

URL: http://10.0.10.22:3001

Current (2026-05-24) — 17 monitors

Name	URL	Email alert
Authentic	https://auth.levkin.ca	✅
Cal.com	https://cal.levkin.ca	✅
Caseware Landing	https://caseware.levkin.ca	✅
Automation Landing	https://auto.levkin.ca	✅
Mailcow	https://mail.levkine.ca	✅
Listmonk	https://listmonk.levkin.ca	✅
Gitea	https://git.levkin.ca	✅
Todo (Vikunja)	https://todo.levkin.ca	✅
Vault	https://vault.levkin.ca	✅
PVENAS	https://10.0.10.10:8006	✅
PVE201	https://10.0.10.201:8006	✅
levkin.ca	https://levkin.ca	✅ added 2026-05-24
Portfolio	https://iliadobkin.com	✅
Search	https://search.levkin.ca	✅
PDF	https://pdf.levkin.ca	✅
Umami script	https://stats.levkin.ca/script.js	✅
Mattermost	https://slack.levkin.ca	✅ (VM 107 — not Hermes)

SMTP notification My Email Alert → alerts@levkine.ca → idobkin@gmail.com. All monitors linked to email alerts.

Public status page: https://status.levkin.ca — slug homelab, domain set in Kuma (17 monitors). Caddy → 10.0.10.22:3001. Admin UI stays at http://10.0.10.22:3001 (LAN only).

Admin password in vault: vault_uptime_kuma_password (admin user).

Add monitors (script)

make vault-export-env   # sets KUMA_PASSWORD
./scripts/kuma-add-monitors.sh

Removed / do not add: Nextcloud (cloud.levkin.ca) — VM 201 retired.

Not in Kuma (by design): Beszel :8090, Dockge :5001, Umami admin :3000 — LAN-only; use Beszel for host metrics.

---

Uptime Kuma — email alerts (Mailcow)

Mail domain is levkine.ca (with e). Cal.com already sends via Mailcow as cal@levkine.ca.

Which email to use

Role	Address	Notes
SMTP server	mail.levkine.ca	Mailcow host
SMTP port	587	STARTTLS (not 465 unless you prefer SMTPS)
From (sender)	alerts@levkine.ca	Create mailbox in Mailcow if it does not exist
To (you)	idobkin@gmail.com or ilia@levkine.ca	Use whichever you read; Gmail is fine for alerts

1. Create mailbox in Mailcow (if needed)

Automated (needs Mailcow API key):

# Define mailbox in group_vars/all/mailcow.yml, password in vault:
make mailcow-mailbox MAILBOX=alerts
# (alias: make mailcow-create-alerts)

# Import from .env into vault once, then delete .env:
cp .env.example .env   # MAILCOW_API_KEY=... ALERTS_PASSWORD=...
make vault-import-env
make mailcow-mailbox MAILBOX=alerts

To add another mailbox tomorrow: edit mailcow.yml + vault_mailcow_mailbox_passwords.<name>, then make mailcow-mailbox MAILBOX=<name>.

Manual UI:

1. https://mail.levkine.ca → admin login
2. Email → Mailboxes → Add → alerts@levkine.ca (strong password → store in Vaultwarden)
3. Optional: alias monitoring@levkine.ca → same inbox

2. Add notification in Kuma

Automated (from your Mac, after mailbox exists):

cd /path/to/ansible
pip install uptime-kuma-api   # or: .venv/bin/pip install uptime-kuma-api
export KUMA_URL=http://10.0.10.22:3001 KUMA_USER=admin KUMA_PASSWORD='...'
export SMTP_USER=alerts@levkine.ca SMTP_PASS='...' SMTP_TO=idobkin@gmail.com
./scripts/kuma-setup-smtp.sh

Manual UI:

1. http://10.0.10.22:3001 → Settings → Notifications → Setup Notification

2. Type: Email (SMTP)

3. Fill in:

   Field	Value
   SMTP Host	mail.levkine.ca
   SMTP Port	587
   Security	TLS / STARTTLS
   Username	alerts@levkine.ca
   Password	mailbox password
   From Email	alerts@levkine.ca
   To Email	idobkin@gmail.com (or your @levkine.ca)

4. Test → save

5. Edit each monitor (or default) → Notifications → enable this channel

Alternative: Mattermost webhook (slack.levkin.ca) if you prefer chat over email.

---

Dockge — what to do after login

On server today:

Path	Contents
/opt/monitoring/compose.yml	Live stack (Docker project monitoring, 4 containers running)
/opt/stacks/monitoring/compose.yaml	Copy for Dockge (same services)
/opt/stacks/authentik-ref/, cal-ref/	README only — no compose file (ignore)

Why “Scan Stacks Folder” looks empty

• Scan only picks up folders under /opt/stacks that contain compose.yaml / compose.yml.
• Your containers were started from /opt/monitoring, so Docker does not automatically link them to /opt/stacks/monitoring until you register that folder in Dockge.

Fix (pick one):

Dockge UI note (your version)

Settings → General only has hostname — there is no “Stacks directory” field. That path is fixed at deploy time:

DOCKGE_STACKS_DIR=/opt/stacks (already set in /opt/monitoring/compose.yml).

Stacks are managed from the home / dashboard page, not Settings.

Option 1 — Add stack manually (recommended)

1. http://10.0.10.22:5001 → home (logo / dashboard, not Settings)
2. + Create Stack (or Compose → new stack)
3. Name: monitoring
4. Path: /opt/stacks/monitoring (must contain compose.yaml)
5. Open stack → review compose → do not Start until old project is stopped (below)

Option 2 — Scan from dashboard menu

1. Stay on dashboard (not Settings)
2. Top-right ⋮ → Scan Stacks Folder
3. Pick monitoring if it appears (authentik-ref / cal-ref have no compose — ignore)

Avoid duplicate containers

Before starting from Dockge:

ssh root@10.0.10.22
cd /opt/monitoring && docker compose down
# Then start from Dockge UI on stack monitoring, OR:
cd /opt/stacks/monitoring && docker compose --env-file .env up -d

Until you do that, Kuma/Dockge/Umami keep running from /opt/monitoring; Dockge is optional for edits until cutover.

Optional reference stacks (read-only)

Create empty stacks under /opt/stacks/ only if you want a UI placeholder:

ssh root@10.0.10.22
mkdir -p /opt/stacks/authentik /opt/stacks/cal
# Copy compose for reference (does NOT control remote host):
scp root@10.0.10.21:/opt/authentik/compose.yml /opt/stacks/authentik/

To manage Authentik or Cal from Dockge long term, either move compose to 218 (not recommended) or install Dockge on each LXC later.

Step 3 — Retire Portainer

VM 109 (portainer) was removed from pve10 on 2026-05-23; use Dockge on 218 instead.

---

Umami

• ✅ Running at http://10.0.10.22:3000 (LAN / Tailscale only)
• ✅ Public tracking via https://stats.levkin.ca/script.js on levkin.ca (LXC 220), caseware, auto, and iliadobkin.com (portfolio LXC 219)

Three choices (pick one later; none block the sites):

Option	Effort	Notes
A — Skip public analytics	0	Use Umami dashboard on :3000 when you care; no DNS/Caddy
B — One DNS + Caddy block	~10 min	A record → home IP + Caddy reverse_proxy 10.0.10.22:3000 on caddy VM
C — Re-add script tags	2 min	After B works, insert script before </head> on 215/216

Suggested public hostname (instead of analytics): stats.levkin.ca (short, clear). Alternatives: umami.levkin.ca, metrics.levkin.ca.

stats.levkin.ca {
    import security-headers
    encode gzip
    reverse_proxy 10.0.10.22:3000
}

Script tag then: https://stats.levkin.ca/script.js

We are not stuck — marketing sites do not need Umami to render. Option A is fine for now.

---

Maintenance

ssh root@10.0.10.22
cd /opt/monitoring
docker compose --env-file .env pull
docker compose --env-file .env up -d
docker compose ps