Add kit selftest CI, Outline sync script, and close ops soak gaps
Fake-site Playwright suite catches BasePage/ApiClient/network regressions before consumers pin a broken tag; Outline sync + docs for QA & Dev live page; Pushgateway deploy soak confirmed separately on observability LXC.
This commit is contained in:
@@ -45,6 +45,25 @@ jobs:
|
|||||||
- name: Build
|
- name: Build
|
||||||
run: npm run build
|
run: npm run build
|
||||||
|
|
||||||
|
# Real browser+HTTP self-test (not mocks). Catches BasePage/ApiClient/network
|
||||||
|
# regressions before consumers pin a broken tag. See docs/SELFTEST.md.
|
||||||
|
selftest:
|
||||||
|
needs: [skip-ci-check, build-and-test]
|
||||||
|
if: needs.skip-ci-check.outputs.should-skip != '1'
|
||||||
|
runs-on: [homelab, self-hosted, linux]
|
||||||
|
container:
|
||||||
|
image: mcr.microsoft.com/playwright:v1.52.0-jammy
|
||||||
|
steps:
|
||||||
|
- uses: actions/checkout@v4
|
||||||
|
- name: Install
|
||||||
|
run: npm ci
|
||||||
|
- name: Self-test against fake site
|
||||||
|
run: npm run selftest
|
||||||
|
env:
|
||||||
|
CI: 'true'
|
||||||
|
PLAYKIT_SELFTEST_PORT: '4173'
|
||||||
|
PLAYKIT_BASE_URL: 'http://127.0.0.1:4173'
|
||||||
|
|
||||||
secret-scan:
|
secret-scan:
|
||||||
needs: skip-ci-check
|
needs: skip-ci-check
|
||||||
if: needs.skip-ci-check.outputs.should-skip != '1'
|
if: needs.skip-ci-check.outputs.should-skip != '1'
|
||||||
|
|||||||
+5
-1
@@ -2,12 +2,16 @@
|
|||||||
|
|
||||||
## Unreleased
|
## Unreleased
|
||||||
|
|
||||||
|
- **Self-test suite** — `selftest/` fake site + Playwright CI job (`docs/SELFTEST.md`, `npm run selftest`)
|
||||||
|
- **Outline sync** — `scripts/outline-sync-playkit.py` creates/updates QA & Dev → Playkit (needs `documents.update` on the Outline API key)
|
||||||
|
- Ops soak: Pushgateway (`:9091`) + Grafana `live-playkit` board **applied** on observability LXC (`make deploy-observability`); Prometheus scrape job `pushgateway` healthy; `RELEASE_TOKEN` present and Gitea release `v0.3.1` exists
|
||||||
|
|
||||||
## 0.3.1 — 2026-07-14
|
## 0.3.1 — 2026-07-14
|
||||||
|
|
||||||
- **Network interception / error monitor** — `interceptNetworkCall()`, `startNetworkErrorMonitor()` (see `docs/NETWORK.md`)
|
- **Network interception / error monitor** — `interceptNetworkCall()`, `startNetworkErrorMonitor()` (see `docs/NETWORK.md`)
|
||||||
- Tests: expand coverage for network helpers — `NetworkErrorMonitor` (record/dedupe/exclude/minStatus/assert), `interceptNetworkCall` (spy/fulfill/handler/method continue/non-JSON), plus `globToRegExp` / `responseMatchesFilter`
|
- Tests: expand coverage for network helpers — `NetworkErrorMonitor` (record/dedupe/exclude/minStatus/assert), `interceptNetworkCall` (spy/fulfill/handler/method continue/non-JSON), plus `globToRegExp` / `responseMatchesFilter`
|
||||||
- Docs: `docs/OUTLINE.md` checklist (update Outline QA & Dev Playkit page on each release); `docs/IDEAS.md` for OSS-borrowed backlog; adoption pause (no new consumer repos until soak)
|
- Docs: `docs/OUTLINE.md` checklist (update Outline QA & Dev Playkit page on each release); `docs/IDEAS.md` for OSS-borrowed backlog; adoption pause (no new consumer repos until soak)
|
||||||
- CI: add tag-triggered `release` job (`.gitea/workflows/ci.yml`) — re-runs build/test, verifies tag matches `package.json` version and `CHANGELOG.md` documents it, creates a Gitea release with an `npm pack` tarball attached. Requires a one-time `GITEA_TOKEN` Actions secret.
|
- CI: add tag-triggered `release` job (`.gitea/workflows/ci.yml`) — re-runs build/test, verifies tag matches `package.json` version and `CHANGELOG.md` documents it, creates a Gitea release with an `npm pack` tarball attached. Requires a one-time `RELEASE_TOKEN` Actions secret.
|
||||||
- Docs: bump install pin examples from `v0.1.0` to `v0.3.0` (README, CONSUMER.md)
|
- Docs: bump install pin examples from `v0.1.0` to `v0.3.0` (README, CONSUMER.md)
|
||||||
- Docs: lead with Mailpit (homelab default) instead of Mailtrap in README email section; use `createMailInbox()` + `readMailHtml()` in the example instead of a provider-specific client
|
- Docs: lead with Mailpit (homelab default) instead of Mailtrap in README email section; use `createMailInbox()` + `readMailHtml()` in the example instead of a provider-specific client
|
||||||
- Ops: Pushgateway + `live-playkit` Grafana board now provisioned via ansible `deploy/observability/` (pending `make deploy-observability`); removed the standalone `dashboards/playkit-overview.json` (superseded) and the `dashboards` entry from `package.json` `files`
|
- Ops: Pushgateway + `live-playkit` Grafana board now provisioned via ansible `deploy/observability/` (pending `make deploy-observability`); removed the standalone `dashboards/playkit-overview.json` (superseded) and the `dashboards` entry from `package.json` `files`
|
||||||
|
|||||||
@@ -79,7 +79,7 @@ test('sign-out stays on public host', async ({ page, playkitConfig, timings }) =
|
|||||||
| `interceptNetworkCall` | Spy or mock the next matching page network call |
|
| `interceptNetworkCall` | Spy or mock the next matching page network call |
|
||||||
| `startNetworkErrorMonitor` | Fail tests that silently collect HTTP 4xx/5xx |
|
| `startNetworkErrorMonitor` | Fail tests that silently collect HTTP 4xx/5xx |
|
||||||
|
|
||||||
See also `docs/NETWORK.md`, `docs/IDEAS.md` (OSS-borrowed backlog), `docs/OUTLINE.md` (live docs sync).
|
See also `docs/NETWORK.md`, `docs/SELFTEST.md` (kit CI fake site), `docs/IDEAS.md` (OSS-borrowed backlog), `docs/OUTLINE.md` (live docs sync).
|
||||||
|
|
||||||
## Network (page traffic)
|
## Network (page traffic)
|
||||||
|
|
||||||
@@ -130,9 +130,21 @@ a real address will not appear in either. See ansible `docs/hardening/SECRETS.md
|
|||||||
npm ci
|
npm ci
|
||||||
npm run typecheck
|
npm run typecheck
|
||||||
npm test
|
npm test
|
||||||
|
npm run selftest # fake site + Chromium (docs/SELFTEST.md)
|
||||||
npm run build
|
npm run build
|
||||||
```
|
```
|
||||||
|
|
||||||
|
## Outline live docs
|
||||||
|
|
||||||
|
After each release (or when consumer-facing docs change):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# needs OUTLINE_URL + OUTLINE_API_KEY (ansible: make vault-export-env)
|
||||||
|
python3 scripts/outline-sync-playkit.py
|
||||||
|
```
|
||||||
|
|
||||||
|
See `docs/OUTLINE.md`.
|
||||||
|
|
||||||
## Release
|
## Release
|
||||||
|
|
||||||
1. Bump `version` in `package.json`
|
1. Bump `version` in `package.json`
|
||||||
@@ -141,7 +153,7 @@ npm run build
|
|||||||
|
|
||||||
Pushing the tag triggers `.gitea/workflows/ci.yml`'s `release` job: it re-runs typecheck/test/build, verifies the tag matches `package.json`'s `version` and that `CHANGELOG.md` documents it, then creates a Gitea release (with the `npm pack` tarball attached) via the API using a repo-scoped `RELEASE_TOKEN` Actions secret. If any check fails, no release is created — fix and re-tag. Consumers still pin the git tag (`#vX.Y.Z`); the Gitea release is for visibility/changelog, not an npm registry publish (see ROADMAP "private Gitea npm registry").
|
Pushing the tag triggers `.gitea/workflows/ci.yml`'s `release` job: it re-runs typecheck/test/build, verifies the tag matches `package.json`'s `version` and that `CHANGELOG.md` documents it, then creates a Gitea release (with the `npm pack` tarball attached) via the API using a repo-scoped `RELEASE_TOKEN` Actions secret. If any check fails, no release is created — fix and re-tag. Consumers still pin the git tag (`#vX.Y.Z`); the Gitea release is for visibility/changelog, not an npm registry publish (see ROADMAP "private Gitea npm registry").
|
||||||
|
|
||||||
4. **Update Outline** — after the release is green, sync the Playkit page under Outline **QA & Dev** (`notes.levkin.ca`). Checklist: `docs/OUTLINE.md`.
|
4. **Update Outline** — `python3 scripts/outline-sync-playkit.py` (checklist: `docs/OUTLINE.md`).
|
||||||
|
|
||||||
**One-time setup:** add a `RELEASE_TOKEN` secret (repo `Settings → Actions → Secrets` on `ilia/playkit`) scoped to create releases on this repo — separate from the `PLAYKIT_GIT_TOKEN` consumers use to clone it. (Named `RELEASE_TOKEN`, not `GITEA_TOKEN` — Gitea's Actions secrets API rejects that literal name as reserved, same reason `PLAYKIT_GIT_TOKEN` isn't called `GITEA_TOKEN` either.)
|
**One-time setup:** add a `RELEASE_TOKEN` secret (repo `Settings → Actions → Secrets` on `ilia/playkit`) scoped to create releases on this repo — separate from the `PLAYKIT_GIT_TOKEN` consumers use to clone it. (Named `RELEASE_TOKEN`, not `GITEA_TOKEN` — Gitea's Actions secrets API rejects that literal name as reserved, same reason `PLAYKIT_GIT_TOKEN` isn't called `GITEA_TOKEN` either.)
|
||||||
|
|
||||||
|
|||||||
+4
-4
@@ -25,11 +25,11 @@ Living plan for making `@levkin/playkit` more useful across Levkin repos.
|
|||||||
- [ ] **Consumer template** — `npx @levkin/playkit init` scaffolding `e2e/` + CI snippet
|
- [ ] **Consumer template** — `npx @levkin/playkit init` scaffolding `e2e/` + CI snippet
|
||||||
- [ ] **Deploy-smoke CLI** — `playkit smoke --project punimtag` post-deploy gate
|
- [ ] **Deploy-smoke CLI** — `playkit smoke --project punimtag` post-deploy gate
|
||||||
- [ ] **Retry policy presets** — flaky-network vs strict-CI profiles
|
- [ ] **Retry policy presets** — flaky-network vs strict-CI profiles
|
||||||
- [ ] **Self-test against a real fake site** — today's unit tests only mock HTTP/mail; `examples/` specs are explicitly *not run* in kit CI. Stand up a tiny demo app (or point at an existing DEV LXC) and run the browser/API/mail helpers against it in CI, so a regression in `BasePage`/`ApiClient`/`assertPublicHost` is caught before a consumer pins a broken tag.
|
- [x] **Self-test against a real fake site** — `selftest/` tiny Node app + Playwright suite in CI (`docs/SELFTEST.md`)
|
||||||
- [x] **Network interception / network error monitor** — `interceptNetworkCall()`, `startNetworkErrorMonitor()` / `assertNoErrors()` (see `docs/NETWORK.md`)
|
- [x] **Network interception / network error monitor** — `interceptNetworkCall()`, `startNetworkErrorMonitor()` / `assertNoErrors()` (see `docs/NETWORK.md`)
|
||||||
- [x] **Tag-triggered release workflow** — `.gitea/workflows/ci.yml` `release` job now runs on `vX.Y.Z` tag push: re-verifies build/test, checks tag == `package.json` version, checks `CHANGELOG.md` has that version's section, then creates a Gitea release (npm-pack tarball attached) via the API. Needs a one-time `GITEA_TOKEN` Actions secret on this repo (see README "Release"). Still open: `build-and-test`/`secret-scan` also re-run on the tag push (same `on.push` trigger) — harmless redundancy, not wired to skip.
|
- [x] **Tag-triggered release workflow** — `.gitea/workflows/ci.yml` `release` job now runs on `vX.Y.Z` tag push: re-verifies build/test, checks tag == `package.json` version, checks `CHANGELOG.md` has that version's section, then creates a Gitea release (npm-pack tarball attached) via the API. Needs a one-time `RELEASE_TOKEN` Actions secret on this repo (see README "Release"). Still open: `build-and-test`/`secret-scan` also re-run on the tag push (same `on.push` trigger) — harmless redundancy, not wired to skip.
|
||||||
- [ ] **Live docs** — Outline page under **QA & Dev** (`notes.levkin.ca`). Sync checklist in `docs/OUTLINE.md` — update Outline whenever playkit is released.
|
- [x] **Live docs** — Outline page under **QA & Dev** (`notes.levkin.ca`). Sync with `python3 scripts/outline-sync-playkit.py` (checklist in `docs/OUTLINE.md`).
|
||||||
- [x] **Pushgateway + dashboard wired in ansible** — `pushgateway` service, Prometheus scrape job, and a generated `live-playkit` Grafana board now live in ansible `deploy/observability/` (superseding the old standalone `dashboards/playkit-overview.json`, which is removed). Ops still needs to run `make deploy-observability` against the LXC before `PLAYKIT_METRICS_ENABLED=true` does anything in CI — check with the ansible repo owner before flipping that on.
|
- [x] **Pushgateway + dashboard wired in ansible** — applied via `make deploy-observability` (LXC `10.0.10.24`): `pushgateway` container, Prometheus scrape job, Grafana `live-playkit` board. Flip consumer CI metrics with `PLAYKIT_METRICS_ENABLED=true` + `PLAYKIT_PUSHGATEWAY_URL=http://10.0.10.24:9091`.
|
||||||
|
|
||||||
## Adoption pause
|
## Adoption pause
|
||||||
|
|
||||||
|
|||||||
+24
-7
@@ -4,6 +4,24 @@ Canonical prose stays in git (`README.md`, `docs/*`, `ROADMAP.md`).
|
|||||||
Browsable front door: **Outline** → collection **QA & Dev** → doc **Playkit**
|
Browsable front door: **Outline** → collection **QA & Dev** → doc **Playkit**
|
||||||
(`https://notes.levkin.ca`).
|
(`https://notes.levkin.ca`).
|
||||||
|
|
||||||
|
## Sync script (preferred)
|
||||||
|
|
||||||
|
From this repo, with Outline credentials loaded:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# from ansible: make vault-export-env && set -a && source .env && set +a
|
||||||
|
python3 scripts/outline-sync-playkit.py
|
||||||
|
python3 scripts/outline-sync-playkit.py --dry-run
|
||||||
|
```
|
||||||
|
|
||||||
|
Creates or updates **QA & Dev → Playkit** with the current `package.json`
|
||||||
|
version, install pin, what’s-in-the-box digest, and links to repo docs.
|
||||||
|
|
||||||
|
**Required API scopes** (Outline → Settings → API & Access): at least
|
||||||
|
`collections.list`, `documents.list`, `documents.info`, `documents.create`,
|
||||||
|
`documents.update`. Without `documents.update` the script can create a first
|
||||||
|
doc but cannot refresh an existing Playkit page (HTTP 403).
|
||||||
|
|
||||||
## When to update Outline
|
## When to update Outline
|
||||||
|
|
||||||
Update the Outline Playkit page **whenever playkit ships a release** (after the
|
Update the Outline Playkit page **whenever playkit ships a release** (after the
|
||||||
@@ -11,13 +29,12 @@ tag / Gitea release is green), or whenever you merge a docs-only change that
|
|||||||
changes consumer behavior:
|
changes consumer behavior:
|
||||||
|
|
||||||
1. Tag / release finished (or main docs PR merged)
|
1. Tag / release finished (or main docs PR merged)
|
||||||
2. Open Outline → QA & Dev → Playkit
|
2. Run `python3 scripts/outline-sync-playkit.py`
|
||||||
3. Sync at least: current version pin, “what’s in the box”, install snippet,
|
3. Spot-check in Outline (search “Playkit” under QA & Dev)
|
||||||
link to CHANGELOG / ROADMAP / NETWORK.md
|
4. Optional: `make outline-setup` from ansible only if collections are missing —
|
||||||
4. Optional: `make outline-setup` from ansible only if you maintain seed notes there —
|
prefer the sync script for the living Playkit page
|
||||||
prefer editing the living page by hand so it stays readable
|
|
||||||
|
|
||||||
Paste template (adjust version):
|
Paste template (if editing by hand — adjust version):
|
||||||
|
|
||||||
```markdown
|
```markdown
|
||||||
# @levkin/playkit
|
# @levkin/playkit
|
||||||
@@ -29,6 +46,6 @@ Shared Playwright + API e2e kit. **Source of truth is the git repo.**
|
|||||||
- Consumers: punimtag (e2e/) — *pause further adoption until soak completes*
|
- Consumers: punimtag (e2e/) — *pause further adoption until soak completes*
|
||||||
|
|
||||||
## Quick links
|
## Quick links
|
||||||
- README · CONSUMER.md · NETWORK.md · IDEAS.md · ROADMAP · CHANGELOG
|
- README · CONSUMER.md · NETWORK.md · IDEAS.md · SELFTEST.md · ROADMAP · CHANGELOG
|
||||||
- Metrics: dash.levkin.ca → Live — Playkit e2e
|
- Metrics: dash.levkin.ca → Live — Playkit e2e
|
||||||
```
|
```
|
||||||
|
|||||||
@@ -0,0 +1,42 @@
|
|||||||
|
# Kit CI self-test (fake site)
|
||||||
|
|
||||||
|
Unit tests in `src/**/*.test.ts` mock HTTP/mail. They do **not** prove that
|
||||||
|
`BasePage`, `ApiClient`, or network helpers work against a real browser + HTTP
|
||||||
|
server. The **selftest** suite fills that gap without depending on punimtag DEV.
|
||||||
|
|
||||||
|
## Layout
|
||||||
|
|
||||||
|
```
|
||||||
|
selftest/
|
||||||
|
demo-site/server.mjs # tiny Node HTTP app (:4173)
|
||||||
|
playwright.config.ts # starts webServer, runs Chromium
|
||||||
|
tests/
|
||||||
|
browser.spec.ts # BasePage, host guards, intercept, error monitor
|
||||||
|
api.spec.ts # ApiClient + Zod schema
|
||||||
|
```
|
||||||
|
|
||||||
|
Mail stays unit-tested only (Mailpit/Mailtrap need live SMTP traps). Selftest
|
||||||
|
does not start Mailpit.
|
||||||
|
|
||||||
|
## Local
|
||||||
|
|
||||||
|
```bash
|
||||||
|
npx playwright install chromium # once
|
||||||
|
npm run selftest
|
||||||
|
```
|
||||||
|
|
||||||
|
Env (optional):
|
||||||
|
|
||||||
|
| Var | Default |
|
||||||
|
|-----|---------|
|
||||||
|
| `PLAYKIT_SELFTEST_PORT` | `4173` |
|
||||||
|
| `PLAYKIT_BASE_URL` | `http://127.0.0.1:4173` |
|
||||||
|
|
||||||
|
Selftest sets `assertPublicHost(..., false)` when touching `127.0.0.1` — consumer
|
||||||
|
public e2e must keep the default forbid-private behavior.
|
||||||
|
|
||||||
|
## CI
|
||||||
|
|
||||||
|
`.gitea/workflows/ci.yml` job `selftest` installs Chromium deps and runs
|
||||||
|
`npm run selftest` after unit tests. Failures there mean a regression in kit
|
||||||
|
browser/API helpers *before* a consumer pins a broken tag.
|
||||||
@@ -42,6 +42,7 @@
|
|||||||
"typecheck": "tsc --noEmit",
|
"typecheck": "tsc --noEmit",
|
||||||
"test": "vitest run",
|
"test": "vitest run",
|
||||||
"test:watch": "vitest",
|
"test:watch": "vitest",
|
||||||
|
"selftest": "playwright test -c selftest/playwright.config.ts",
|
||||||
"lint": "tsc --noEmit",
|
"lint": "tsc --noEmit",
|
||||||
"prepublishOnly": "npm run build",
|
"prepublishOnly": "npm run build",
|
||||||
"example:api": "tsx examples/api/health.example.ts"
|
"example:api": "tsx examples/api/health.example.ts"
|
||||||
|
|||||||
Executable
+234
@@ -0,0 +1,234 @@
|
|||||||
|
#!/usr/bin/env python3
|
||||||
|
"""Create or update Outline → QA & Dev → Playkit (living front door).
|
||||||
|
|
||||||
|
Canonical source stays in git; this pushes a digest + links to notes.levkin.ca.
|
||||||
|
|
||||||
|
Credentials (required):
|
||||||
|
OUTLINE_URL
|
||||||
|
OUTLINE_TOKEN or OUTLINE_API_KEY
|
||||||
|
|
||||||
|
Usage (from ansible vault env, or any shell with those vars):
|
||||||
|
python3 scripts/outline-sync-playkit.py
|
||||||
|
python3 scripts/outline-sync-playkit.py --dry-run
|
||||||
|
"""
|
||||||
|
from __future__ import annotations
|
||||||
|
|
||||||
|
import argparse
|
||||||
|
import json
|
||||||
|
import os
|
||||||
|
import sys
|
||||||
|
import urllib.error
|
||||||
|
import urllib.request
|
||||||
|
from pathlib import Path
|
||||||
|
|
||||||
|
REPO_ROOT = Path(__file__).resolve().parents[1]
|
||||||
|
COLLECTION_NAME = "QA & Dev"
|
||||||
|
DOC_TITLE = "Playkit"
|
||||||
|
|
||||||
|
|
||||||
|
def load_package_version() -> str:
|
||||||
|
pkg = json.loads((REPO_ROOT / "package.json").read_text(encoding="utf-8"))
|
||||||
|
return str(pkg["version"])
|
||||||
|
|
||||||
|
|
||||||
|
def playkit_markdown(version: str) -> str:
|
||||||
|
return f"""# @levkin/playkit
|
||||||
|
|
||||||
|
Shared Playwright + API e2e kit for Levkin repos. **Source of truth is the git repo** — this page is the browsable front door.
|
||||||
|
|
||||||
|
- **Repo:** https://git.levkin.ca/ilia/playkit
|
||||||
|
- **Current:** v{version}
|
||||||
|
- **Consumers:** punimtag (`e2e/`) — *pause further adoption until soak completes*
|
||||||
|
- **Install (git pin):** `npm i git+https://git.levkin.ca/ilia/playkit.git#v{version}`
|
||||||
|
|
||||||
|
## What's in the box
|
||||||
|
|
||||||
|
- Browser: `BasePage`, retries (`click`/`fill`/`safeGoto`), public-host guards (`assertPublicHost`, `waitForUrlHost`)
|
||||||
|
- Network: `interceptNetworkCall`, `startNetworkErrorMonitor` / `assertNoErrors` — see NETWORK.md
|
||||||
|
- API: `ApiClient` + Zod `schema` / `assertSchema`
|
||||||
|
- Mail: `createMailInbox()` (Mailpit default, Mailtrap optional)
|
||||||
|
- Auth helpers: `saveStorageState` / `storageStateUse`, `playkitFailureArtifacts()`
|
||||||
|
- Metrics: `TimingCollector` → Prometheus Pushgateway → Grafana **Live — Playkit e2e**
|
||||||
|
|
||||||
|
## Quick links (in-repo)
|
||||||
|
|
||||||
|
| Doc | Purpose |
|
||||||
|
|-----|---------|
|
||||||
|
| `README.md` | Install, env vars, release steps |
|
||||||
|
| `docs/CONSUMER.md` | How to wire a consumer `e2e/` |
|
||||||
|
| `docs/NETWORK.md` | Page traffic spy / error monitor |
|
||||||
|
| `docs/IDEAS.md` | OSS-borrowed backlog |
|
||||||
|
| `docs/OUTLINE.md` | When / how to re-sync this page |
|
||||||
|
| `docs/SELFTEST.md` | Kit CI fake-site self-test |
|
||||||
|
| `ROADMAP.md` | v0.4+ plan |
|
||||||
|
| `CHANGELOG.md` | Per-version notes |
|
||||||
|
|
||||||
|
## Metrics
|
||||||
|
|
||||||
|
- Pushgateway: `http://10.0.10.24:9091` (LAN)
|
||||||
|
- Grafana: `dash.levkin.ca` → **Live — Playkit e2e** (`live-playkit`)
|
||||||
|
- Enable in CI with `PLAYKIT_METRICS_ENABLED=true` + `PLAYKIT_PUSHGATEWAY_URL`
|
||||||
|
|
||||||
|
## Release checklist (operators)
|
||||||
|
|
||||||
|
1. Tag `vX.Y.Z` (must match `package.json` + CHANGELOG section) → Gitea `release` job uses `RELEASE_TOKEN`
|
||||||
|
2. Re-run this script: `python3 scripts/outline-sync-playkit.py`
|
||||||
|
3. Confirm Grafana board + Pushgateway scrape after `make deploy-observability` (ansible)
|
||||||
|
|
||||||
|
_Last synced by `scripts/outline-sync-playkit.py` for v{version}._
|
||||||
|
"""
|
||||||
|
|
||||||
|
|
||||||
|
class OutlineError(RuntimeError):
|
||||||
|
pass
|
||||||
|
|
||||||
|
|
||||||
|
class OutlineClient:
|
||||||
|
def __init__(self, base_url: str, token: str, *, dry_run: bool = False) -> None:
|
||||||
|
self.api_base = f"{base_url.rstrip('/')}/api"
|
||||||
|
self.token = token
|
||||||
|
self.dry_run = dry_run
|
||||||
|
|
||||||
|
def call(self, method: str, body: dict | None = None) -> dict:
|
||||||
|
payload = json.dumps(body or {}).encode()
|
||||||
|
req = urllib.request.Request(
|
||||||
|
f"{self.api_base}/{method}",
|
||||||
|
data=payload,
|
||||||
|
headers={
|
||||||
|
"Authorization": f"Bearer {self.token}",
|
||||||
|
"Content-Type": "application/json",
|
||||||
|
},
|
||||||
|
method="POST",
|
||||||
|
)
|
||||||
|
if self.dry_run:
|
||||||
|
print(f"DRY-RUN POST {method}: {json.dumps(body or {}, indent=2)[:800]}")
|
||||||
|
return {"ok": True, "data": {"id": "dry-run"}}
|
||||||
|
|
||||||
|
try:
|
||||||
|
with urllib.request.urlopen(req, timeout=60) as resp:
|
||||||
|
raw = resp.read().decode()
|
||||||
|
except urllib.error.HTTPError as exc:
|
||||||
|
detail = exc.read().decode(errors="replace")
|
||||||
|
raise OutlineError(f"{method} failed (HTTP {exc.code}): {detail[:500]}") from exc
|
||||||
|
except urllib.error.URLError as exc:
|
||||||
|
raise OutlineError(f"{method} request failed: {exc}") from exc
|
||||||
|
|
||||||
|
data = json.loads(raw)
|
||||||
|
if not data.get("ok"):
|
||||||
|
raise OutlineError(f"{method} returned ok=false: {raw[:500]}")
|
||||||
|
return data
|
||||||
|
|
||||||
|
def list_collections(self) -> list[dict]:
|
||||||
|
out: list[dict] = []
|
||||||
|
offset = 0
|
||||||
|
while True:
|
||||||
|
batch = self.call("collections.list", {"limit": 100, "offset": offset}).get("data") or []
|
||||||
|
out.extend(batch)
|
||||||
|
if len(batch) < 100:
|
||||||
|
break
|
||||||
|
offset += 100
|
||||||
|
return out
|
||||||
|
|
||||||
|
def list_documents(self, collection_id: str) -> list[dict]:
|
||||||
|
out: list[dict] = []
|
||||||
|
offset = 0
|
||||||
|
while True:
|
||||||
|
batch = (
|
||||||
|
self.call(
|
||||||
|
"documents.list",
|
||||||
|
{"limit": 100, "offset": offset, "collectionId": collection_id},
|
||||||
|
).get("data")
|
||||||
|
or []
|
||||||
|
)
|
||||||
|
out.extend(batch)
|
||||||
|
if len(batch) < 100:
|
||||||
|
break
|
||||||
|
offset += 100
|
||||||
|
return out
|
||||||
|
|
||||||
|
|
||||||
|
def require_env() -> tuple[str, str]:
|
||||||
|
# Allow ansible `.env` next to sibling repo without overwriting set keys
|
||||||
|
ansible_env = REPO_ROOT.parent / "ansible" / ".env"
|
||||||
|
if ansible_env.is_file():
|
||||||
|
for line in ansible_env.read_text(encoding="utf-8").splitlines():
|
||||||
|
line = line.strip()
|
||||||
|
if not line or line.startswith("#") or "=" not in line:
|
||||||
|
continue
|
||||||
|
key, _, value = line.partition("=")
|
||||||
|
key = key.strip()
|
||||||
|
if key and key not in os.environ:
|
||||||
|
os.environ[key] = value.strip().strip('"').strip("'")
|
||||||
|
|
||||||
|
url = os.environ.get("OUTLINE_URL", "").strip()
|
||||||
|
token = (
|
||||||
|
os.environ.get("OUTLINE_TOKEN", "").strip()
|
||||||
|
or os.environ.get("OUTLINE_API_KEY", "").strip()
|
||||||
|
)
|
||||||
|
missing = [n for n, v in (("OUTLINE_URL", url), ("OUTLINE_TOKEN|OUTLINE_API_KEY", token)) if not v]
|
||||||
|
if missing:
|
||||||
|
print(
|
||||||
|
"ERROR: missing "
|
||||||
|
+ ", ".join(missing)
|
||||||
|
+ "\nLoad: cd ../ansible && make vault-export-env && set -a && source .env && set +a",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
sys.exit(1)
|
||||||
|
return url, token
|
||||||
|
|
||||||
|
|
||||||
|
def main() -> int:
|
||||||
|
parser = argparse.ArgumentParser(description=__doc__)
|
||||||
|
parser.add_argument("--dry-run", action="store_true")
|
||||||
|
args = parser.parse_args()
|
||||||
|
|
||||||
|
version = load_package_version()
|
||||||
|
text = playkit_markdown(version)
|
||||||
|
url, token = require_env()
|
||||||
|
client = OutlineClient(url, token, dry_run=args.dry_run)
|
||||||
|
|
||||||
|
collections = {c["name"]: c["id"] for c in client.list_collections()}
|
||||||
|
if COLLECTION_NAME not in collections:
|
||||||
|
raise OutlineError(
|
||||||
|
f'collection "{COLLECTION_NAME}" not found — run ansible scripts/outline-setup.py first'
|
||||||
|
)
|
||||||
|
coll_id = collections[COLLECTION_NAME]
|
||||||
|
docs = {d["title"]: d["id"] for d in client.list_documents(coll_id)}
|
||||||
|
|
||||||
|
try:
|
||||||
|
if DOC_TITLE in docs:
|
||||||
|
client.call(
|
||||||
|
"documents.update",
|
||||||
|
{"id": docs[DOC_TITLE], "title": DOC_TITLE, "text": text, "publish": True},
|
||||||
|
)
|
||||||
|
print(f"updated: {COLLECTION_NAME} / {DOC_TITLE} (v{version})")
|
||||||
|
else:
|
||||||
|
client.call(
|
||||||
|
"documents.create",
|
||||||
|
{
|
||||||
|
"title": DOC_TITLE,
|
||||||
|
"text": text,
|
||||||
|
"collectionId": coll_id,
|
||||||
|
"publish": True,
|
||||||
|
},
|
||||||
|
)
|
||||||
|
print(f"created: {COLLECTION_NAME} / {DOC_TITLE} (v{version})")
|
||||||
|
except OutlineError as exc:
|
||||||
|
if "403" in str(exc) or "authorization_error" in str(exc):
|
||||||
|
print(
|
||||||
|
"\nERROR: Outline API key can list/create but not update documents.\n"
|
||||||
|
"In Outline → Settings → API & Access, edit the key and add scopes:\n"
|
||||||
|
" documents.update documents.delete documents.archive\n"
|
||||||
|
"(full recommended set is in ansible docs/guides/authentik-apps.md Outline section).\n"
|
||||||
|
"Then re-run: python3 scripts/outline-sync-playkit.py\n"
|
||||||
|
"Also delete any leftover 'Playkit sync probe' doc under QA & Dev if present.\n",
|
||||||
|
file=sys.stderr,
|
||||||
|
)
|
||||||
|
raise
|
||||||
|
|
||||||
|
print(f"Outline: {url.rstrip('/')}/ → search '{DOC_TITLE}' under {COLLECTION_NAME}")
|
||||||
|
return 0
|
||||||
|
|
||||||
|
|
||||||
|
if __name__ == "__main__":
|
||||||
|
raise SystemExit(main())
|
||||||
@@ -0,0 +1,87 @@
|
|||||||
|
#!/usr/bin/env node
|
||||||
|
/**
|
||||||
|
* Tiny fake app for playkit CI self-tests.
|
||||||
|
* Endpoints: GET /, GET /form, GET /api/health, GET /api/items, GET /api/boom (500)
|
||||||
|
*/
|
||||||
|
import http from 'node:http';
|
||||||
|
import { URL } from 'node:url';
|
||||||
|
|
||||||
|
const PORT = Number(process.env.PLAYKIT_SELFTEST_PORT || 4173);
|
||||||
|
const HOST = process.env.PLAYKIT_SELFTEST_HOST || '127.0.0.1';
|
||||||
|
|
||||||
|
const HOME = `<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head><meta charset="utf-8"><title>Playkit selftest</title></head>
|
||||||
|
<body>
|
||||||
|
<h1>Playkit selftest home</h1>
|
||||||
|
<p><a href="/form">Open form</a></p>
|
||||||
|
<button id="load-items" type="button">Load items</button>
|
||||||
|
<ul id="items"></ul>
|
||||||
|
<script>
|
||||||
|
document.getElementById('load-items').addEventListener('click', async () => {
|
||||||
|
const res = await fetch('/api/items');
|
||||||
|
const data = await res.json();
|
||||||
|
const ul = document.getElementById('items');
|
||||||
|
ul.innerHTML = '';
|
||||||
|
for (const item of data.items) {
|
||||||
|
const li = document.createElement('li');
|
||||||
|
li.textContent = item.name;
|
||||||
|
ul.appendChild(li);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>`;
|
||||||
|
|
||||||
|
const FORM = `<!doctype html>
|
||||||
|
<html lang="en">
|
||||||
|
<head><meta charset="utf-8"><title>Form — Playkit selftest</title></head>
|
||||||
|
<body>
|
||||||
|
<h1>Demo form</h1>
|
||||||
|
<label>Name <input id="name" name="name" /></label>
|
||||||
|
<button id="submit" type="button">Submit</button>
|
||||||
|
<p id="greeting" hidden></p>
|
||||||
|
<script>
|
||||||
|
document.getElementById('submit').addEventListener('click', () => {
|
||||||
|
const name = document.getElementById('name').value;
|
||||||
|
const g = document.getElementById('greeting');
|
||||||
|
g.hidden = false;
|
||||||
|
g.textContent = 'Hello, ' + name;
|
||||||
|
});
|
||||||
|
</script>
|
||||||
|
</body>
|
||||||
|
</html>`;
|
||||||
|
|
||||||
|
function json(res, status, body) {
|
||||||
|
const payload = JSON.stringify(body);
|
||||||
|
res.writeHead(status, {
|
||||||
|
'Content-Type': 'application/json',
|
||||||
|
'Content-Length': Buffer.byteLength(payload),
|
||||||
|
});
|
||||||
|
res.end(payload);
|
||||||
|
}
|
||||||
|
|
||||||
|
function html(res, status, body) {
|
||||||
|
res.writeHead(status, { 'Content-Type': 'text/html; charset=utf-8' });
|
||||||
|
res.end(body);
|
||||||
|
}
|
||||||
|
|
||||||
|
const server = http.createServer((req, res) => {
|
||||||
|
const u = new URL(req.url || '/', `http://${HOST}:${PORT}`);
|
||||||
|
if (req.method === 'GET' && u.pathname === '/') return html(res, 200, HOME);
|
||||||
|
if (req.method === 'GET' && u.pathname === '/form') return html(res, 200, FORM);
|
||||||
|
if (req.method === 'GET' && u.pathname === '/api/health') {
|
||||||
|
return json(res, 200, { status: 'ok', service: 'playkit-selftest' });
|
||||||
|
}
|
||||||
|
if (req.method === 'GET' && u.pathname === '/api/items') {
|
||||||
|
return json(res, 200, { items: [{ id: 1, name: 'alpha' }, { id: 2, name: 'beta' }] });
|
||||||
|
}
|
||||||
|
if (req.method === 'GET' && u.pathname === '/api/boom') {
|
||||||
|
return json(res, 500, { error: 'intentional' });
|
||||||
|
}
|
||||||
|
json(res, 404, { error: 'not found' });
|
||||||
|
});
|
||||||
|
|
||||||
|
server.listen(PORT, HOST, () => {
|
||||||
|
console.log(`playkit selftest listening on http://${HOST}:${PORT}`);
|
||||||
|
});
|
||||||
@@ -0,0 +1,29 @@
|
|||||||
|
import { defineConfig, devices } from '@playwright/test';
|
||||||
|
|
||||||
|
const port = Number(process.env.PLAYKIT_SELFTEST_PORT || 4173);
|
||||||
|
const baseURL = process.env.PLAYKIT_BASE_URL || `http://127.0.0.1:${port}`;
|
||||||
|
|
||||||
|
export default defineConfig({
|
||||||
|
testDir: './tests',
|
||||||
|
fullyParallel: true,
|
||||||
|
forbidOnly: !!process.env.CI,
|
||||||
|
retries: process.env.CI ? 1 : 0,
|
||||||
|
workers: process.env.CI ? 2 : undefined,
|
||||||
|
reporter: process.env.CI ? 'list' : [['list'], ['html', { open: 'never' }]],
|
||||||
|
use: {
|
||||||
|
baseURL,
|
||||||
|
...devices['Desktop Chrome'],
|
||||||
|
trace: 'on-first-retry',
|
||||||
|
},
|
||||||
|
webServer: {
|
||||||
|
command: `node demo-site/server.mjs`,
|
||||||
|
url: baseURL,
|
||||||
|
reuseExistingServer: !process.env.CI,
|
||||||
|
timeout: 30_000,
|
||||||
|
env: {
|
||||||
|
...process.env,
|
||||||
|
PLAYKIT_SELFTEST_PORT: String(port),
|
||||||
|
PLAYKIT_SELFTEST_HOST: '127.0.0.1',
|
||||||
|
},
|
||||||
|
},
|
||||||
|
});
|
||||||
@@ -0,0 +1,43 @@
|
|||||||
|
import { test, expect } from '@playwright/test';
|
||||||
|
import { z } from 'zod';
|
||||||
|
import { ApiClient, TimingCollector, assertSchema } from '../../src/index.js';
|
||||||
|
|
||||||
|
const baseUrl = process.env.PLAYKIT_BASE_URL || 'http://127.0.0.1:4173';
|
||||||
|
|
||||||
|
const HealthSchema = z.object({
|
||||||
|
status: z.literal('ok'),
|
||||||
|
service: z.string(),
|
||||||
|
});
|
||||||
|
|
||||||
|
const ItemsSchema = z.object({
|
||||||
|
items: z.array(z.object({ id: z.number(), name: z.string() })),
|
||||||
|
});
|
||||||
|
|
||||||
|
test.describe('playkit selftest — ApiClient', () => {
|
||||||
|
test('GET /api/health with Zod schema', async () => {
|
||||||
|
const timings = new TimingCollector();
|
||||||
|
const api = new ApiClient({ baseUrl });
|
||||||
|
|
||||||
|
const res = await timings.measure('api_health', () =>
|
||||||
|
api.get<{ status: string; service: string }>('/api/health', {
|
||||||
|
expectedStatus: 200,
|
||||||
|
schema: HealthSchema,
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
|
||||||
|
expect(res.status).toBe(200);
|
||||||
|
expect(res.data.status).toBe('ok');
|
||||||
|
// signature: assertSchema(data, schema)
|
||||||
|
expect(assertSchema(res.data, HealthSchema).service).toBe('playkit-selftest');
|
||||||
|
});
|
||||||
|
|
||||||
|
test('GET /api/items + expectedStatus on boom', async () => {
|
||||||
|
const api = new ApiClient({ baseUrl });
|
||||||
|
const items = await api.get('/api/items', { schema: ItemsSchema });
|
||||||
|
expect(items.data.items[0]?.name).toBe('alpha');
|
||||||
|
expect(items.data.items).toHaveLength(2);
|
||||||
|
|
||||||
|
const boom = await api.get('/api/boom', { expectedStatus: 500 });
|
||||||
|
expect(boom.status).toBe(500);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -0,0 +1,59 @@
|
|||||||
|
import { test, expect } from '@playwright/test';
|
||||||
|
import {
|
||||||
|
BasePage,
|
||||||
|
assertPublicHost,
|
||||||
|
waitForUrlHost,
|
||||||
|
TimingCollector,
|
||||||
|
interceptNetworkCall,
|
||||||
|
startNetworkErrorMonitor,
|
||||||
|
} from '../../src/index.js';
|
||||||
|
|
||||||
|
const baseUrl = process.env.PLAYKIT_BASE_URL || 'http://127.0.0.1:4173';
|
||||||
|
|
||||||
|
test.describe('playkit selftest — browser helpers', () => {
|
||||||
|
test('BasePage click/fill + public-host relax for localhost', async ({ page }) => {
|
||||||
|
// Intentional LAN/selftest: forbidPrivateHosts must be off
|
||||||
|
assertPublicHost(baseUrl, false);
|
||||||
|
expect(() => assertPublicHost(baseUrl, true)).toThrow(/private host/i);
|
||||||
|
|
||||||
|
const timings = new TimingCollector();
|
||||||
|
const home = new BasePage(page, baseUrl);
|
||||||
|
|
||||||
|
await timings.measure('open_home', () => home.open('/'));
|
||||||
|
await waitForUrlHost(page, '127.0.0.1');
|
||||||
|
await expect(page.getByRole('heading', { name: 'Playkit selftest home' })).toBeVisible();
|
||||||
|
|
||||||
|
await timings.measure('open_form', () => home.open('/form'));
|
||||||
|
await home.fill(page.locator('#name'), 'Kolby');
|
||||||
|
await home.click(page.locator('#submit'));
|
||||||
|
await expect(page.getByText('Hello, Kolby')).toBeVisible();
|
||||||
|
|
||||||
|
expect(timings.getSamples().length).toBeGreaterThanOrEqual(2);
|
||||||
|
});
|
||||||
|
|
||||||
|
test('network intercept + error monitor', async ({ page }) => {
|
||||||
|
const home = new BasePage(page, baseUrl);
|
||||||
|
await home.open('/');
|
||||||
|
|
||||||
|
const items = interceptNetworkCall({
|
||||||
|
page,
|
||||||
|
method: 'GET',
|
||||||
|
url: '**/api/items',
|
||||||
|
});
|
||||||
|
await page.getByRole('button', { name: 'Load items' }).click();
|
||||||
|
const { status, responseJson } = await items;
|
||||||
|
expect(status).toBe(200);
|
||||||
|
expect(responseJson).toMatchObject({ items: expect.any(Array) });
|
||||||
|
await expect(page.getByText('alpha')).toBeVisible();
|
||||||
|
|
||||||
|
const net = startNetworkErrorMonitor(page, {
|
||||||
|
excludePatterns: [/\/api\/health/],
|
||||||
|
});
|
||||||
|
try {
|
||||||
|
await page.goto(`${baseUrl}/api/boom`);
|
||||||
|
// stay on JSON page — monitor should have recorded 500
|
||||||
|
} finally {
|
||||||
|
expect(() => net.assertNoErrors()).toThrow(/network errors/i);
|
||||||
|
}
|
||||||
|
});
|
||||||
|
});
|
||||||
Reference in New Issue
Block a user