Create the setup for the documentation page (#171)

* don't run for generated docusaurus * format * workflow to autoupdate docs version * fix versioning * add link back to app * remove old docs * html link??? * don't track .docusaurus * documantation build
2026-02-15 22:50:52 +00:00
parent 3c41df9ba8
commit 1f929dfc7f
76 changed files with 105 additions and 2786 deletions
@@ -0,0 +1,46 @@
+---
+id: gradcracker
+title: Gradcracker Extractor
+description: How the Gradcracker crawler builds search URLs and extracts jobs.
+sidebar_position: 2
+---
+
+A plain-English walkthrough of the Gradcracker extractor in `extractors/gradcracker`.
+
+## Big picture
+
+The crawler builds search URLs, scrapes listing pages, then opens job details for descriptions and apply URLs.
+
+## 1) Build search URLs
+
+- Combines UK regions with role terms.
+- Defaults include roles such as `web-development` and `software-systems`.
+- `GRADCRACKER_SEARCH_TERMS` overrides defaults.
+
+## 2) Crawl list pages
+
+- Waits for job cards (`article[wire:key]`).
+- Extracts title, employer, discipline, deadline, salary, location, degree, start date.
+- Queues job detail pages.
+
+Controls:
+
+- `GRADCRACKER_MAX_JOBS_PER_TERM`
+- `JOBOPS_SKIP_APPLY_FOR_EXISTING=1`
+- `JOBOPS_EXISTING_JOB_URLS` / `JOBOPS_EXISTING_JOB_URLS_FILE`
+
+## 3) Crawl detail pages
+
+- Waits for `.body-content`
+- Captures full description text
+- Clicks apply button to resolve final application URL
+- Handles popup and same-tab redirects
+
+## 4) Progress reporting
+
+Set `JOBOPS_EMIT_PROGRESS=1` for structured progress lines consumable by orchestrator UI.
+
+## Notes
+
+- Uses Playwright + Crawlee via Camoufox.
+- Low concurrency and longer timeouts for stability.
@@ -0,0 +1,47 @@
+---
+id: jobspy
+title: JobSpy Extractor
+description: How the JobSpy Python wrapper is orchestrated and normalized.
+sidebar_position: 3
+---
+
+A walkthrough of the JobSpy extractor for Indeed, LinkedIn, and Glassdoor.
+
+## Big picture
+
+JobSpy runs as a Python script per search term, writes JSON, then orchestrator ingests and normalizes into internal job shape.
+
+## 1) Inputs and defaults
+
+Key environment variables:
+
+- `JOBSPY_SITES` (default: `indeed,linkedin`)
+- `JOBSPY_SEARCH_TERM` (default: `web developer`)
+- `JOBSPY_LOCATION` (default: `UK`)
+- `JOBSPY_RESULTS_WANTED` (default: `200`)
+- `JOBSPY_HOURS_OLD` (default: `72`)
+- `JOBSPY_COUNTRY_INDEED` (default: `UK`)
+- `JOBSPY_LINKEDIN_FETCH_DESCRIPTION` (default: `true`)
+
+## 2) Orchestrator flow
+
+The service in `orchestrator/src/server/services/jobspy.ts`:
+
+- Builds search-term list from UI or env
+- Runs Python once per term with unique output file
+- Reads JSON and maps to `CreateJobInput`
+- De-dupes by `jobUrl`
+- Deletes temp output files best-effort
+
+## 3) Mapping and cleanup
+
+- Normalizes salary ranges
+- Converts empty values to null
+- Keeps metadata like skills, ratings, remote flags when available
+- Skips rows with invalid site or missing URL
+
+## Notes
+
+- `JOBSPY_SEARCH_TERMS` can be JSON array or `|`, comma, newline-delimited text.
+- Set `JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0` to speed runs.
+- Temp output files are stored under `data/imports/`.
@@ -0,0 +1,51 @@
+---
+id: manual
+title: Manual Import Extractor
+description: Import jobs from pasted descriptions and run AI-assisted inference.
+sidebar_position: 4
+---
+
+Manual import lets users add jobs that automated scrapers miss.
+
+## Big picture
+
+User pastes raw description, AI infers structure, user reviews edits, then import saves and scores the job.
+
+## 1) Input
+
+User pastes a job description in the **Manual Import** UI.
+
+## 2) AI inference
+
+Endpoint:
+
+- `POST /api/manual-jobs/infer`
+
+Service:
+
+- `orchestrator/src/server/services/manualJob.ts`
+
+Behavior:
+
+- Sends raw text to configured LLM
+- Extracts structured fields (title, employer, location, salary, etc.)
+- Returns inferred JSON for user review
+
+If no LLM key is configured, inference is skipped and user can fill fields manually.
+
+## 3) Review and edit
+
+User reviews inferred fields and corrects missing/wrong values.
+
+## 4) Storage and scoring
+
+Import endpoint:
+
+- `POST /api/manual-jobs/import`
+
+On import:
+
+- Generates unique job ID if URL absent
+- Stores source as `manual`
+- Triggers async suitability scoring
+- Persists score and reason
@@ -0,0 +1,13 @@
+---
+id: overview
+title: Extractors Overview
+description: Technical index of supported extractors and how they work.
+sidebar_position: 1
+---
+
+Technical breakdowns of each extractor:
+
+- [Gradcracker](/docs/extractors/gradcracker)
+- [JobSpy](/docs/extractors/jobspy)
+- [UKVisaJobs](/docs/extractors/ukvisajobs)
+- [Manual Import](/docs/extractors/manual)
@@ -0,0 +1,73 @@
+---
+id: ukvisajobs
+title: UKVisaJobs Extractor
+description: Authenticated session flow, API pagination, and orchestrator ingestion.
+sidebar_position: 5
+---
+
+UKVisaJobs is the most complex extractor because authenticated sessions are required.
+
+## Big picture
+
+Two layers:
+
+1. `extractors/ukvisajobs/src/main.ts` handles login/API calls and dataset output.
+2. `orchestrator/src/server/services/ukvisajobs.ts` executes extractor and ingests/de-dupes output.
+
+## 1) Authentication and session cache
+
+Session cache file:
+
+- `extractors/ukvisajobs/storage/ukvisajobs-auth.json`
+
+Flow:
+
+- Reuse cached token/cookies when valid
+- Re-login with Playwright + Camoufox when needed
+- Refresh and retry on token-expired responses
+
+Force refresh:
+
+- `UKVISAJOBS_REFRESH_ONLY=1`
+
+## 2) API requests
+
+Endpoint:
+
+- `https://my.ukvisajobs.com/ukvisa-api/api/fetch-jobs-data`
+
+Each request includes auth token + session cookies and paginates (15 jobs/page).
+
+## 3) Mapping
+
+- Normalizes salary from min/max/interval
+- Builds fallback visa description when content missing
+- Maps `job_link` to both `jobUrl` and `applicationLink`
+
+## 4) Output dataset
+
+Written to:
+
+- `extractors/ukvisajobs/storage/datasets/default/`
+
+Includes per-job JSON files and combined `jobs.json`.
+
+## 5) Orchestrator flow
+
+- Spawns extractor (`npx tsx src/main.ts`)
+- Runs terms sequentially with delay
+- De-dupes by `sourceJobId` (fallback `jobUrl`)
+- Fetches detail pages when descriptions are too short
+
+## Controls
+
+- `UKVISAJOBS_EMAIL`, `UKVISAJOBS_PASSWORD`
+- `UKVISAJOBS_HEADLESS`
+- `UKVISAJOBS_MAX_JOBS` (default 50, max 200)
+- `UKVISAJOBS_SEARCH_KEYWORD`
+
+## Practical notes
+
+- Deleting auth cache forces next run to re-login.
+- Low-concurrency/polite scraping by design.
+- If extractor breaks, check session refresh path first.
@@ -0,0 +1,61 @@
+---
+id: ghostwriter
+title: Ghostwriter
+description: Context-aware per-job AI chat assistant behavior and API surface.
+sidebar_position: 2
+---
+
+Ghostwriter is the per-job AI chat assistant in JobOps.
+
+## What it is for
+
+Ghostwriter uses:
+
+- Current job description and metadata
+- Reduced profile snapshot
+- Global writing style settings
+
+Typical use cases:
+
+- Role-specific answer drafting
+- Cover letter and outreach drafts
+- Interview prep tied to the JD
+- Rephrasing with tone constraints
+
+## Where it appears
+
+- Available from job details in `discovered` and `ready`
+- Right-side drawer UX
+- One persistent conversation per job
+
+## Writing style settings impact
+
+Global settings affecting generations:
+
+- `Tone`
+- `Formality`
+- `Constraints`
+- `Do-not-use terms`
+
+Defaults:
+
+- Tone: `professional`
+- Formality: `medium`
+- Constraints: empty
+- Do-not-use terms: empty
+
+## Context and safety model
+
+- Job snapshot is truncated to fit prompt budget.
+- Profile snapshot includes relevant slices only.
+- System prompt enforces read-only assistant behavior.
+- Logging stores metadata, not full prompt/response dumps.
+
+## API surface
+
+- `GET /api/jobs/:id/chat/messages`
+- `POST /api/jobs/:id/chat/messages` (streaming)
+- `POST /api/jobs/:id/chat/runs/:runId/cancel`
+- `POST /api/jobs/:id/chat/messages/:assistantMessageId/regenerate` (streaming)
+
+Compatibility thread endpoints remain, but UI behavior is one thread per job.
@@ -0,0 +1,92 @@
+---
+id: orchestrator
+title: Orchestrator
+description: Job states, ready flow, and PDF generation/regeneration behavior.
+sidebar_position: 1
+---
+
+This guide explains job states, how jobs become ready, and how PDF generation works.
+
+## Job states
+
+- `discovered`: Found by crawler/import, not tailored yet.
+- `processing`: Tailoring and/or PDF generation in progress.
+- `ready`: Tailored PDF generated and ready to apply.
+- `applied`: Marked as applied.
+- `skipped`: Explicitly excluded from active queue.
+- `expired`: Deadline passed.
+
+## Intended ready flow
+
+### 1) Manual flow
+
+1. Job starts in `discovered`.
+2. Open in Discovered panel and choose Tailor.
+3. Edit JD/tailored fields/project picks.
+4. Click **Finalize & Move to Ready**.
+
+### 2) Auto flow
+
+1. Pipeline scores discovered jobs.
+2. Top jobs above threshold are auto-processed.
+3. Jobs move directly to `ready` with generated PDFs.
+
+## Ghostwriter
+
+Ghostwriter is available in `discovered` and `ready` job views.
+
+For details, see [Ghostwriter](/docs/features/ghostwriter).
+
+## Generating PDFs (first time)
+
+PDF generation uses:
+
+- Base resume selected from RxResume
+- Job description
+- Tailored summary/headline/skills/projects
+
+Common paths:
+
+- **Discovered → Tailor → Finalize**
+  - `POST /api/jobs/:id/process`
+- **Ready → Regenerate PDF**
+  - `POST /api/jobs/:id/generate-pdf`
+
+## Regenerating PDFs after edits
+
+If JD or tailoring changes, regenerate PDF to keep output in sync.
+
+### API flow
+
+```bash
+PATCH /api/jobs/:id
+{
+  "jobDescription": "<new JD>",
+  "tailoredSummary": "<optional>",
+  "tailoredHeadline": "<optional>",
+  "tailoredSkills": "[{\"name\":\"Backend\",\"keywords\":[\"TypeScript\",\"Node.js\"]}]",
+  "selectedProjectIds": "p1,p2"
+}
+```
+
+```bash
+POST /api/jobs/:id/summarize?force=true
+POST /api/jobs/:id/generate-pdf
+```
+
+## Post-application tracking
+
+For inbox routing flow and setup, see [Post-Application Tracking](/docs/features/post-application-tracking).
+
+## Notes and gotchas
+
+- `processing` is transient. On PDF failure, job reverts to `discovered`.
+- PDFs are served at `/pdfs/resume_<jobId>.pdf` with cache-bust on `updatedAt`.
+- `skipped`/`applied` jobs can be reopened by patching `status` to `discovered`.
+
+## External payload and sanitization defaults
+
+- LLM prompts send minimized profile/job fields.
+- Webhooks are sanitized and whitelisted by default.
+- Logs and error details are redacted/truncated by default.
+- Correlation fields include `requestId`, and when available `pipelineRunId` and `jobId`.
@@ -0,0 +1,86 @@
+---
+id: post-application-tracking
+title: Post-Application Tracking
+description: Gmail-based tracking inbox, smart routing, and review workflow.
+sidebar_position: 3
+---
+
+The Tracking Inbox monitors Gmail for job-application responses and updates timelines.
+
+## Overview
+
+1. Scans Gmail for recruitment-related emails
+2. Matches emails to tracked jobs using AI
+3. Updates timeline/state when confidence is high
+4. Queues uncertain matches for manual review
+
+## Smart router flow
+
+```mermaid
+flowchart TD
+    A[Recruitment email arrives in Gmail] --> B[Smart Router AI analyzes content]
+    B --> C{How confident is the match?}
+
+    C -->|95-100%| D[Auto-linked to job]
+    D --> E[Timeline updated automatically]
+
+    C -->|50-94%| F[Goes to Inbox for review with suggested match]
+
+    C -->|<50%| G{Is it relevant?}
+    G -->|Yes| H[Goes to Inbox as orphan]
+    G -->|No| I[Ignored]
+```
+
+## Setup
+
+### Prerequisites
+
+1. Gmail account with application emails
+2. Google OAuth credentials
+
+### Configure OAuth
+
+Set:
+
+```bash
+GMAIL_OAUTH_CLIENT_ID=your-client-id.apps.googleusercontent.com
+GMAIL_OAUTH_CLIENT_SECRET=your-client-secret
+GMAIL_OAUTH_REDIRECT_URI=https://your-domain.com/oauth/gmail/callback
+```
+
+Then connect in UI via **Tracking Inbox → Connect Gmail**.
+
+## Using the inbox
+
+- Review pending items in Tracking Inbox
+- Approve to link/update timeline
+- Ignore to mark non-relevant
+
+Confidence interpretation:
+
+- `95-100%`: auto-processed
+- `50-94%`: pending review with suggestion
+- `<50%`: pending review as orphan/ignored
+
+## Privacy and security
+
+- Scope requested: `gmail.readonly`
+- Minimal metadata sent for matching
+- Email data stays local in your instance
+
+## API reference
+
+| Method | Endpoint                                  | Description           |
+| ------ | ----------------------------------------- | --------------------- |
+| GET    | `/api/post-application/inbox`             | List pending messages |
+| POST   | `/api/post-application/inbox/:id/approve` | Approve message       |
+| POST   | `/api/post-application/inbox/:id/deny`    | Ignore message        |
+| GET    | `/api/post-application/runs`              | List sync runs        |
+| POST   | `/api/post-application/gmail/connect`     | Start OAuth flow      |
+| GET    | `/api/post-application/gmail/callback`    | OAuth callback        |
+
+## Common issues
+
+- No refresh token: disconnect and reconnect Gmail.
+- Emails not appearing: check runs, OAuth config, and recruitment subjects.
+- Wrong matches: expected in lower-confidence buckets; use manual review.
@@ -0,0 +1,122 @@
+---
+id: self-hosting
+title: Self-Hosting (Docker Compose)
+description: Deploy JobOps with Docker Compose and configure onboarding integrations.
+sidebar_position: 1
+---
+
+The easiest way to run JobOps is via Docker Compose. The app is self-configuring and guides you through setup on first launch.
+
+## Prerequisites
+
+- Docker Desktop or Docker Engine + Compose v2
+
+## 1) Start the stack
+
+No environment variables are required to boot:
+
+```bash
+docker compose up -d
+```
+
+This pulls the pre-built image from GHCR and starts the API, UI, and scrapers in one container.
+
+To build locally instead:
+
+```bash
+docker compose up -d --build
+```
+
+## 2) Access the app and onboard
+
+Open:
+
+- **Dashboard**: `http://localhost:3005`
+
+The onboarding wizard helps you validate and save:
+
+1. **LLM Provider**: OpenRouter by default (or OpenAI/Gemini/local URL).
+2. **PDF Export**: RxResume credentials for PDF generation.
+3. **Template Resume**: Choose a base resume from your RxResume account.
+
+Settings are saved to the local database.
+
+## Gmail OAuth (Tracking Inbox)
+
+If you want Gmail integration, configure OAuth credentials.
+
+### 1) Create Google OAuth credentials
+
+In Google Cloud:
+
+1. Configure OAuth consent screen.
+2. Enable Gmail API.
+3. Create OAuth client ID (`Web application`).
+4. Add redirect URI:
+  - `http://localhost:3005/oauth/gmail/callback`
+  - Or your production URL, for example `https://your-domain.com/oauth/gmail/callback`
+
+### 2) Configure environment variables
+
+- `GMAIL_OAUTH_CLIENT_ID` (required)
+- `GMAIL_OAUTH_CLIENT_SECRET` (required)
+- `GMAIL_OAUTH_REDIRECT_URI` (optional, recommended in production)
+
+### 3) Restart and connect
+
+- Restart container
+- Open Tracking Inbox and click **Connect Gmail**
+
+## Email-to-job matching overview
+
+```mermaid
+flowchart TD
+    A[Recruitment email arrives in Gmail] --> B[Smart Router AI analyzes content]
+    B --> C{How confident is the match?}
+
+    C -->|95-100%| D[Auto-linked to job]
+    D --> E[Timeline updated automatically]
+
+    C -->|50-94%| F[Goes to Inbox for review with suggested job match]
+
+    C -->|<50%| G{Is it relevant?}
+    G -->|Yes| H[Goes to Inbox as orphan]
+    G -->|No| I[Ignored]
+
+    F --> J{User review}
+    H --> J
+    J -->|Approve| K[Linked to job + timeline update]
+    J -->|Ignore| L[Marked not relevant]
+```
+
+## Persistent data
+
+`./data` bind-mount stores:
+
+- SQLite DB: `data/jobs.db`
+- Generated PDFs: `data/pdfs/`
+
+## Public demo mode
+
+Set `DEMO_MODE=true` for sandbox deployments.
+
+Behavior in demo mode:
+
+- Works locally: browsing/filtering/status/timeline edits
+- Simulated: pipeline run/summarize/process/rescore/pdf/apply
+- Blocked: settings writes, DB clear, backups
+- Auto-reset: every 6 hours
+
+## Updating
+
+```bash
+git pull
+docker compose pull
+docker compose up -d
+```
+
+## Self-hosted Reactive Resume
+
+If you self-host Reactive Resume, set:
+
+- `RXRESUME_URL=http://rxresume.local.net`
@@ -0,0 +1,77 @@
+---
+id: intro
+title: JobOps Documentation
+description: Documentation index for setup, features, extractors, and common problems.
+sidebar_position: 1
+slug: /
+---
+
+Welcome to the JobOps documentation. This site contains guides for setup, configuration, and day-to-day usage.
+
+## Getting Started
+
+- **[Self-Hosting Guide](/docs/getting-started/self-hosting)**
+  - Docker setup instructions
+  - Gmail OAuth configuration for email tracking
+  - Environment variables reference
+  - Demo mode deployment
+
+## Feature Documentation
+
+- **[Orchestrator](/docs/features/orchestrator)**
+  - Job states explained (`discovered`, `ready`, `applied`, etc.)
+  - The ready flow (manual vs auto)
+  - PDF generation and regeneration
+  - Post-application tracking overview
+
+- **[Ghostwriter](/docs/features/ghostwriter)**
+  - One persistent conversation per job
+  - Streaming responses, stop, and regenerate
+  - Markdown rendering and drawer behavior
+  - Writing style settings impact
+
+- **[Post-Application Tracking](/docs/features/post-application-tracking)**
+  - How the Smart Router AI works
+  - Gmail integration setup
+  - Using the Tracking Inbox
+  - Privacy and security details
+  - API reference
+
+## Extractors
+
+- **[Extractors Overview](/docs/extractors/overview)**
+- **[Gradcracker](/docs/extractors/gradcracker)**
+- **[UKVisaJobs](/docs/extractors/ukvisajobs)**
+- **[JobSpy](/docs/extractors/jobspy)**
+- **[Manual Import](/docs/extractors/manual)**
+
+## Quick Reference
+
+### Main Components
+
+- **Orchestrator**: Main application (UI, API, database)
+- **Extractors**: Specialized job crawlers
+- **Shared**: Common types and utilities
+
+### Key Features
+
+1. **Job Discovery**: Automatically find jobs from multiple sources.
+2. **AI Scoring**: Rank jobs by suitability for your profile.
+3. **Resume Tailoring**: Generate custom resumes for each job.
+4. **PDF Export**: Create tailored PDFs via RxResume integration.
+5. **Application Tracking**: Monitor your applied jobs.
+6. **Email Tracking**: Auto-track post-application responses.
+
+## Contributing to Documentation
+
+When adding user-visible behavior:
+
+1. Update the relevant feature page in current docs.
+2. Add API documentation where relevant.
+3. Keep examples realistic and copy-pasteable.
+4. Include diagrams for non-trivial workflows.
+
+## Support
+
+- Open an [issue](https://github.com/DaKheera47/job-ops/issues) for documentation errors.
+- Check these docs before opening support requests.
@@ -0,0 +1,34 @@
+---
+id: documentation-style-guide
+title: Documentation Style Guide
+description: Standards for writing user-facing docs in this repository.
+sidebar_position: 2
+---
+
+Use this structure for feature pages:
+
+1. **What it is**
+2. **Why it exists**
+3. **How to use it**
+4. **Common problems**
+5. **Related pages**
+
+## Frontmatter requirements
+
+Every doc should include:
+
+- `id`
+- `title`
+- `description`
+- `sidebar_position`
+
+## Writing rules
+
+- Prefer concrete steps over abstract prose.
+- Provide copy-pasteable examples.
+- State defaults and constraints explicitly.
+- Link related docs using `/docs/...` URLs.
+
+## PR expectations
+
+Any user-visible behavior change should include matching docs updates.
@@ -0,0 +1,26 @@
+---
+id: faq
+title: FAQ
+description: Frequently asked questions about deployment, docs, and operations.
+sidebar_position: 1
+---
+
+## Is docs content bundled for self-hosted installs?
+
+Yes. The docs static build is bundled and served locally at `/docs`.
+
+## Do I need internet access to read docs after deployment?
+
+No, for bundled docs pages.
+
+## How are docs versions managed?
+
+Docs are versioned using Docusaurus versions, intended to map to release tags.
+
+## Where should contributors edit docs?
+
+Edit files under `docs-site/docs` for latest docs.
+
+## Are old markdown files in `documentation/` still canonical?
+
+No. `docs-site/docs` is the source of truth after migration.
@@ -0,0 +1,41 @@
+---
+id: common-problems
+title: Common Problems
+description: Quick fixes for the most frequent setup and runtime issues.
+sidebar_position: 1
+---
+
+## Docs site not loading at `/docs`
+
+- Confirm docs build exists:
+
+```bash
+npm --workspace docs-site run build
+```
+
+- In production, ensure container includes docs build artifact.
+
+## Deep links under `/docs/*` return 404
+
+- Confirm Express is serving docs static mount before app SPA fallback.
+- Confirm docs base URL is `/docs/` in `docs-site/docusaurus.config.ts`.
+
+## Gmail OAuth callback fails
+
+- Verify `GMAIL_OAUTH_CLIENT_ID`, `GMAIL_OAUTH_CLIENT_SECRET`.
+- Ensure authorized redirect URI exactly matches deployment callback URL.
+
+## No job scoring or AI inference
+
+- Validate `LLM_API_KEY` and provider settings.
+- Check settings page and API connectivity.
+
+## PDF generation fails
+
+- Verify RxResume credentials.
+- Confirm selected base resume exists and is accessible.
+
+## UKVisaJobs runs fail
+
+- Re-authenticate by removing cached auth file or forcing refresh.
+- Verify extractor credentials and API response behavior.