032626bd7d
* initial * refactor: centralize SSE plumbing for client and server * docs: add centralized SSE usage standards to agents guide * use sse to stream actions to the client * ui: align bulk progress toast with default sonner style * ui: remove hide action from bulk progress toast * full width progress bar * fix(stream): track client disconnect and writability * fix(stream): stop bulk loop when SSE client disconnects * fix(stream): avoid writing error/end to closed SSE response * fix(stream): gate started/progress frames on writable SSE socket * types(api): narrow SSE stream payload input contract * refactor(ui): share clamp helper for bulk progress * fix(stream): add heartbeat to bulk action SSE route * feat(stream): include completed count in bulk completion event * fix(client-sse): separate parse vs handler errors and cancel reader
113 lines
4.3 KiB
Markdown
113 lines
4.3 KiB
Markdown
# Error/Logging/Sanitization Standards
|
|
|
|
This project uses strict operability and privacy defaults for server-side code.
|
|
|
|
## API Response Contract
|
|
|
|
For all `/api/*` routes, return:
|
|
|
|
- Success: `{ ok: true, data, meta?: { requestId } }`
|
|
- Error: `{ ok: false, error: { code, message, details? }, meta: { requestId } }`
|
|
|
|
Use consistent status/code mapping:
|
|
|
|
- `400 INVALID_REQUEST`
|
|
- `401 UNAUTHORIZED`
|
|
- `403 FORBIDDEN`
|
|
- `404 NOT_FOUND`
|
|
- `408 REQUEST_TIMEOUT`
|
|
- `409 CONFLICT`
|
|
- `422 UNPROCESSABLE_ENTITY`
|
|
- `500 INTERNAL_ERROR`
|
|
- `502 UPSTREAM_ERROR`
|
|
- `503 SERVICE_UNAVAILABLE`
|
|
|
|
## Correlation IDs
|
|
|
|
- Honor inbound `x-request-id` when present; otherwise generate one.
|
|
- Always return `x-request-id` header.
|
|
- Include request ID in API responses (`meta.requestId`) and logs.
|
|
- Propagate context into async flows (especially pipeline run and per-job work) so logs include `pipelineRunId` / `jobId` when available.
|
|
|
|
## Logging Rules
|
|
|
|
- Use the shared logger wrapper (`infra/logger.ts`) in core server paths.
|
|
- Do not add direct `console.log`, `console.warn`, or `console.error` in core paths.
|
|
- Log structured objects, not free-form dumps.
|
|
- Include useful context fields (e.g. `requestId`, `pipelineRunId`, `jobId`, `route`, `status`).
|
|
|
|
## SSE Standards
|
|
|
|
- Use centralized SSE helpers by default.
|
|
- Server: use `orchestrator/src/server/infra/sse.ts` for setup, data writes, comments, and heartbeats.
|
|
- Client (`EventSource`): use `orchestrator/src/client/lib/sse.ts` for subscription/open/message/error plumbing.
|
|
- Do not duplicate raw SSE setup (`Content-Type`, `Connection`, heartbeat loops, or ad-hoc `JSON.parse` event parsing) when these helpers apply.
|
|
- Keep feature payload types domain-local (pipeline, ghostwriter, bulk actions), but reuse shared transport plumbing.
|
|
|
|
## Redaction and Sanitization
|
|
|
|
- Always sanitize objects before logging or returning in error `details`.
|
|
- Redact sensitive keys by default (`authorization`, `cookie`, `password`, `secret`, `token`, `apiKey`, etc.).
|
|
- Truncate large payloads and long strings.
|
|
- Do not throw/log raw upstream response bodies, full webhook bodies, or large `JSON.stringify(...)` blobs.
|
|
|
|
## Webhook and LLM Payload Defaults
|
|
|
|
- Webhooks: send minimal whitelisted payloads by default.
|
|
- LLM prompts: send only required profile/job fields; avoid unnecessary PII.
|
|
- Document external payload behavior when adding new integrations.
|
|
|
|
## PR Checklist (Routes/Services)
|
|
|
|
- API responses follow `{ ok, data/error, meta.requestId }`.
|
|
- Status/code mapping is correct and consistent.
|
|
- Request/correlation IDs appear in logs and async workflows.
|
|
- No raw sensitive payload logging or raw upstream body throws.
|
|
- New/changed webhook or LLM payloads are sanitized and documented.
|
|
|
|
## Documentation Standards (Condensed)
|
|
|
|
When adding or updating user-facing docs:
|
|
|
|
- Use this feature-page structure:
|
|
1. **What it is**
|
|
2. **Why it exists**
|
|
3. **How to use it**
|
|
4. **Common problems**
|
|
5. **Related pages**
|
|
- Include frontmatter keys: `id`, `title`, `description`, `sidebar_position`.
|
|
- Prefer concrete, step-by-step instructions over abstract explanation.
|
|
- Include copy-pasteable examples where relevant.
|
|
- State defaults and constraints explicitly.
|
|
- Link related docs with `/docs/...` URLs.
|
|
- Any user-visible behavior change should include corresponding docs updates.
|
|
|
|
## Validation / Verification
|
|
|
|
Before marking work complete, verify changes with the same checks used by CI.
|
|
|
|
### Required CI-parity checks
|
|
|
|
Run from repository root:
|
|
|
|
1. `./orchestrator/node_modules/.bin/biome ci .`
|
|
2. `npm run check:types:shared`
|
|
3. `npm --workspace orchestrator run check:types`
|
|
4. `npm --workspace gradcracker-extractor run check:types`
|
|
5. `npm --workspace ukvisajobs-extractor run check:types`
|
|
6. `npm --workspace orchestrator run build:client`
|
|
7. `npm --workspace orchestrator run test:run`
|
|
|
|
### Native module note (better-sqlite3)
|
|
|
|
If tests fail with a Node ABI mismatch for `better-sqlite3`, rebuild it before running tests:
|
|
|
|
- `npm --workspace orchestrator rebuild better-sqlite3`
|
|
|
|
CI runs on Node 22. If local behavior differs, verify with Node 22 before concluding a change is valid.
|
|
|
|
### Scope-specific checks
|
|
|
|
- For focused changes, run targeted tests first (for touched files/modules), then still run the full CI-parity list above before finalizing.
|
|
- A change is considered valid only when all required checks pass without ignored failures.
|