ilia/Jobber
1
0
Fork 0
Code Issues Pull Requests Actions Packages Projects Releases Wiki Activity

docs: cut version 0.1.29

Browse Source
This commit is contained in:
github-actions[bot]
2026-03-06 12:22:43 +00:00
parent 6a19fff436
commit 2214e6d2cb
32 changed files with 2967 additions and 0 deletions
Download Patch File Download Diff File Expand all files Collapse all files
docs-site/versioned_docs/version-0.1.29/extractors/adzuna.md
+66
View File
@@ -0,0 +1,66 @@
---
id: adzuna
title: Adzuna Extractor
description: API-based Adzuna extraction with orchestrator ingestion and progress updates.
sidebar_position: 6
---
## What it is
Original website: [adzuna.com](https://www.adzuna.com)
Adzuna is an API-backed extractor implemented in two lean pieces:
1. `extractors/adzuna/src/main.ts` fetches paginated Adzuna search results and writes `jobs.json`.
2. `orchestrator/src/server/services/adzuna.ts` runs the extractor, parses progress lines, and maps rows into `CreateJobInput`.
It de-duplicates in the existing repository path using `sourceJobId` fallback to `jobUrl`.
## Why it exists
Adzuna provides stable API discovery for countries that are not covered by UK-only sources. It adds a lower-maintenance source without introducing new API routes or UI sections.
## How to use it
1. Create an Adzuna developer account.
2. Open [Adzuna Access Details](https://developer.adzuna.com/admin/access_details).
3. Copy your **App ID** and **App Key**.
4. In Job Ops, open **Settings** and paste them into `Adzuna App ID` and `Adzuna App Key` under **Environment & Accounts**.
5. In **Pipeline Run** (Automatic tab), select a compatible country and enable **Adzuna** in Sources.
6. Start the run; Adzuna progress appears in the existing crawl progress stream.
City behavior:
- If **Search cities** are set in Automatic advanced settings, Adzuna runs once per city.
- City runs use strict post-filtering (`job.location` contains requested city) to avoid broad country-level spillover.
Default controls:
- `ADZUNA_APP_ID`
- `ADZUNA_APP_KEY`
- `ADZUNA_MAX_JOBS_PER_TERM` (default `50`)
- `ADZUNA_LOCATION_QUERY` (optional city/location text)
Supported countries in this integration:
- United Kingdom, United States, Austria, Australia, Belgium, Brazil, Canada, Switzerland, Germany, Spain, France, India, Italy, Mexico, Netherlands, New Zealand, Poland, Singapore, South Africa.
## Common problems
### Adzuna is disabled in source selection
- `Adzuna App ID` and `Adzuna App Key` are missing from Settings (or env).
### Adzuna is skipped for my selected country
- The selected country is not in the supported list above.
### Adzuna fails with authorization errors
- Verify `ADZUNA_APP_ID` and `ADZUNA_APP_KEY` are valid and active in your Adzuna account.
## Related pages
- [Extractors Overview](/docs/next/extractors/overview)
- [Pipeline Run](/docs/next/features/pipeline-run)
- [Settings](/docs/next/features/settings)
docs-site/versioned_docs/version-0.1.29/extractors/gradcracker.md
+48
View File
@@ -0,0 +1,48 @@
---
id: gradcracker
title: Gradcracker Extractor
description: How the Gradcracker crawler builds search URLs and extracts jobs.
sidebar_position: 2
---
A plain-English walkthrough of the Gradcracker extractor in `extractors/gradcracker`.
Original website: [gradcracker.com](https://www.gradcracker.com)
## Big picture
The crawler builds search URLs, scrapes listing pages, then opens job details for descriptions and apply URLs.
## 1) Build search URLs
- Combines UK regions with role terms.
- Defaults include roles such as `web-development` and `software-systems`.
- `GRADCRACKER_SEARCH_TERMS` overrides defaults.
## 2) Crawl list pages
- Waits for job cards (`article[wire:key]`).
- Extracts title, employer, discipline, deadline, salary, location, degree, start date.
- Queues job detail pages.
Controls:
- `GRADCRACKER_MAX_JOBS_PER_TERM`
- `JOBOPS_SKIP_APPLY_FOR_EXISTING=1`
- `JOBOPS_EXISTING_JOB_URLS` / `JOBOPS_EXISTING_JOB_URLS_FILE`
## 3) Crawl detail pages
- Waits for `.body-content`
- Captures full description text
- Clicks apply button to resolve final application URL
- Handles popup and same-tab redirects
## 4) Progress reporting
Set `JOBOPS_EMIT_PROGRESS=1` for structured progress lines consumable by orchestrator UI.
## Notes
- Uses Playwright + Crawlee via Camoufox.
- Low concurrency and longer timeouts for stability.
docs-site/versioned_docs/version-0.1.29/extractors/hiring-cafe.md
+77
View File
@@ -0,0 +1,77 @@
---
id: hiring-cafe
title: Hiring Cafe Extractor
description: Browser-backed Hiring Cafe extraction integrated into the pipeline source selector.
sidebar_position: 7
---
## What it is
Original website: [hiring.cafe](https://hiring.cafe)
Special thanks: Initial implementation inspiration came from [umur957/hiring-cafe-job-scraper](https://github.com/umur957/hiring-cafe-job-scraper).
Hiring Cafe is a browser-backed extractor that queries Hiring Cafe search APIs and maps results into the orchestrator `CreateJobInput` shape.
Implementation split:
1. `extractors/hiringcafe/src/main.ts` builds search state, calls Hiring Cafe APIs, and writes dataset JSON.
2. `orchestrator/src/server/services/hiring-cafe.ts` runs the extractor, streams progress events, and maps rows for pipeline import.
## Why it exists
Hiring Cafe adds another non-credentialed source that can be enabled from the existing source picker, without adding new settings UI.
It also supports term-by-term search and country-aware search state using the same pipeline knobs you already set for automatic runs.
## How to use it
1. Open **Run jobs** and choose **Automatic**.
2. **Hiring Cafe** is enabled by default in **Sources** (toggle it off if you do not want it for this run).
3. Set your existing automatic run knobs:
- `searchTerms` drive per-term Hiring Cafe `searchQuery`.
- selected country maps into Hiring Cafe location search state.
- run budget path (`jobspyResultsWanted`) is reused as the max jobs-per-term cap.
- optional **Search cities** narrow results by city.
4. Start the run and watch progress in the pipeline progress card.
Defaults and constraints:
- No new Hiring Cafe settings fields were added.
- `worldwide` and `usa/ca` run in broad mode without a strict country location filter.
- Hiring Cafe is enabled by default in source selection.
- `HIRING_CAFE_DATE_FETCHED_PAST_N_DAYS` controls recency window when running extractor directly (default `7`).
- When a city is provided via `searchCities`, Hiring Cafe uses city radius search (default `1` mile) and strict city post-filtering.
- City geocoding is resolved through Nominatim (OpenStreetMap data); if you scale extractor traffic, add attribution and cache repeated city lookups.
Local run example:
```bash
HIRING_CAFE_SEARCH_TERMS='["backend engineer"]' \
HIRING_CAFE_COUNTRY='united kingdom' \
HIRING_CAFE_MAX_JOBS_PER_TERM='50' \
npm --workspace hiringcafe-extractor run start
```
## Common problems
### Hiring Cafe returns 429 / Vercel security checkpoint
- The extractor first attempts Camoufox-backed Firefox and falls back to vanilla Firefox startup if Camoufox is unstable locally.
- If upstream blocks continue, retry later or reduce run concurrency at the pipeline level by selecting fewer sources.
### Hiring Cafe does not appear in sources
- Check that client is running on latest build containing the new source list.
- Hiring Cafe is source-only and does not require credentials, so it should appear once the new build is loaded.
### Results are lower than expected
- Cap is tied to automatic run budget path (`jobspyResultsWanted`) and search term count.
- Country mapping can narrow results when a strict country location is applied.
## Related pages
- [Extractors Overview](/docs/next/extractors/overview)
- [Pipeline Run](/docs/next/features/pipeline-run)
- [Settings](/docs/next/features/settings)
docs-site/versioned_docs/version-0.1.29/extractors/jobspy.md
+52
View File
@@ -0,0 +1,52 @@
---
id: jobspy
title: JobSpy Extractor
description: How the JobSpy Python wrapper is orchestrated and normalized.
sidebar_position: 3
---
A walkthrough of the JobSpy extractor for Indeed, LinkedIn, and Glassdoor.
Original websites:
- [indeed.com](https://www.indeed.com)
- [linkedin.com/jobs](https://www.linkedin.com/jobs)
- [glassdoor.com](https://www.glassdoor.com)
## Big picture
JobSpy runs as a Python script per search term, writes JSON, then orchestrator ingests and normalizes into internal job shape.
## 1) Inputs and defaults
Key environment variables:
- `JOBSPY_SITES` (default: `indeed,linkedin`)
- `JOBSPY_SEARCH_TERM` (default: `web developer`)
- `JOBSPY_LOCATION` (default: `UK`)
- `JOBSPY_RESULTS_WANTED` (default: `200`)
- `JOBSPY_HOURS_OLD` (default: `72`)
- `JOBSPY_COUNTRY_INDEED` (default: `UK`)
- `JOBSPY_LINKEDIN_FETCH_DESCRIPTION` (default: `true`)
## 2) Orchestrator flow
The service in `orchestrator/src/server/services/jobspy.ts`:
- Builds search-term list from UI or env
- Runs Python once per term with unique output file
- Reads JSON and maps to `CreateJobInput`
- De-dupes by `jobUrl`
- Deletes temp output files best-effort
## 3) Mapping and cleanup
- Normalizes salary ranges
- Converts empty values to null
- Keeps metadata like skills, ratings, remote flags when available
- Skips rows with invalid site or missing URL
## Notes
- `JOBSPY_SEARCH_TERMS` can be JSON array or `|`, comma, newline-delimited text.
- Set `JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0` to speed runs.
- Temp output files are stored under `data/imports/`.
docs-site/versioned_docs/version-0.1.29/extractors/manual.md
+61
View File
@@ -0,0 +1,61 @@
---
id: manual
title: Manual Import Extractor
description: Import jobs from pasted descriptions and run AI-assisted inference.
sidebar_position: 4
---
Manual import lets users add jobs that automated scrapers miss.
## Big picture
User pastes raw description, AI infers structure, user reviews edits, then import saves and scores the job.
## 1) Input
Manual import accepts:
- plain text job descriptions
- raw HTML job descriptions
- job links/URLs
When a URL is provided, backend fetch attempts depend on whether the page can be resolved with `curl`. Some job sites block or heavily script content, so certain links will not resolve cleanly.
## 2) AI inference
Endpoint:
- `POST /api/manual-jobs/infer`
Service:
- `orchestrator/src/server/services/manualJob.ts`
Behavior:
- Converts the provided input into text context and sends it to the configured LLM
- Extracts structured fields (title, employer, location, salary, etc.)
- Returns inferred JSON for user review
Practical limit:
- The inference quality ceiling is mostly the configured model capability and context behavior. Better model quality generally yields better field extraction.
If no LLM key is configured, inference is skipped and user can fill fields manually.
## 3) Review and edit
User reviews inferred fields and corrects missing/wrong values.
## 4) Storage and scoring
Import endpoint:
- `POST /api/manual-jobs/import`
On import:
- Generates unique job ID if URL absent
- Stores source as `manual`
- Triggers async suitability scoring
- Persists score and reason
docs-site/versioned_docs/version-0.1.29/extractors/overview.md
+42
View File
@@ -0,0 +1,42 @@
---
id: overview
title: Extractors Overview
description: Technical index of supported extractors and how they work.
sidebar_position: 1
---
This page helps you choose the right extractor for your run, understand key constraints, and navigate to detailed technical guides.
Extractor integrations are now registered through manifests and loaded automatically at orchestrator startup. Runtime discovery only scans `extractors/*/(manifest.ts|src/manifest.ts)` and does not read manifests from `orchestrator/**`. Extractor-specific run logic should also remain in `extractors/<name>/` so orchestrator stays source-agnostic. To add a new source, follow [Add an Extractor](/docs/next/workflows/add-an-extractor).
## Extractor chooser
| Extractor | Best use case | Core constraints/dependencies | Notable controls | Output/behavior notes |
| --- | --- | --- | --- | --- |
| [Gradcracker](/docs/next/extractors/gradcracker) | UK graduate roles from Gradcracker | Crawling stability depends on page structure and anti-bot behavior; tuned for low concurrency | `GRADCRACKER_SEARCH_TERMS`, `GRADCRACKER_MAX_JOBS_PER_TERM`, `JOBOPS_SKIP_APPLY_FOR_EXISTING` | Scrapes listing metadata, then detail pages and apply URL resolution |
| [JobSpy](/docs/next/extractors/jobspy) | Multi-source discovery (Indeed, LinkedIn, Glassdoor) | Requires Python wrapper execution per term; source availability and quality vary by site/location | `JOBSPY_SITES`, `JOBSPY_SEARCH_TERMS`, `JOBSPY_RESULTS_WANTED`, `JOBSPY_HOURS_OLD`, `JOBSPY_LINKEDIN_FETCH_DESCRIPTION` | Produces JSON per term, then orchestrator normalizes and de-duplicates by `jobUrl` |
| [Adzuna](/docs/next/extractors/adzuna) | API-based multi-country discovery with low scraping overhead | Requires valid App ID/App Key; country must be in Adzuna-supported list | `ADZUNA_APP_ID`, `ADZUNA_APP_KEY`, `ADZUNA_MAX_JOBS_PER_TERM` | API pagination to dataset output; orchestrator maps progress and de-duplicates by `sourceJobId`/`jobUrl` |
| [Hiring Cafe](/docs/next/extractors/hiring-cafe) | Browser-backed discovery using Hiring Cafe search APIs | Subject to upstream anti-bot checks; uses browser context and encoded search-state payloads | `HIRING_CAFE_SEARCH_TERMS`, `HIRING_CAFE_COUNTRY`, `HIRING_CAFE_MAX_JOBS_PER_TERM`, `HIRING_CAFE_DATE_FETCHED_PAST_N_DAYS` | Uses existing pipeline term/country/budget knobs and maps directly to normalized jobs |
| [UKVisaJobs](/docs/next/extractors/ukvisajobs) | UK visa sponsorship-focused roles | Requires authenticated session and periodic token/cookie refresh | `UKVISAJOBS_EMAIL`, `UKVISAJOBS_PASSWORD`, `UKVISAJOBS_MAX_JOBS`, `UKVISAJOBS_SEARCH_KEYWORD` | API pagination + dataset output; orchestrator de-dupes and may fetch missing descriptions |
| [Manual Import](/docs/next/extractors/manual) | One-off jobs not covered by scrapers | Inference quality depends on model/provider and input quality; some URLs cannot be fetched reliably | App/API endpoints (`/api/manual-jobs/infer`, `/api/manual-jobs/import`) | Accepts text/HTML/URL, runs inference, then saves and scores job after review |
## Which extractor should I use?
- Use **JobSpy** for broad first-pass sourcing across common boards.
- Use **Adzuna** when you want API-first discovery in supported non-UK markets.
- Use **Hiring Cafe** when you want another term/country-driven source without adding credentials.
- Use **Gradcracker** when targeting graduate pipelines in the UK.
- Use **UKVisaJobs** for sponsorship-specific UK searches.
- Use **Manual Import** when you already have a specific posting and need direct import.
Many runs combine sources: broad discovery first, then manual import for high-priority jobs that scraping misses.
## Related extractor docs
- [Gradcracker](/docs/next/extractors/gradcracker)
- [JobSpy](/docs/next/extractors/jobspy)
- [Adzuna](/docs/next/extractors/adzuna)
- [Hiring Cafe](/docs/next/extractors/hiring-cafe)
- [UKVisaJobs](/docs/next/extractors/ukvisajobs)
- [Manual Import](/docs/next/extractors/manual)
- [Add an Extractor](/docs/next/workflows/add-an-extractor)
docs-site/versioned_docs/version-0.1.29/extractors/ukvisajobs.md
+75
View File
@@ -0,0 +1,75 @@
---
id: ukvisajobs
title: UKVisaJobs Extractor
description: Authenticated session flow, API pagination, and orchestrator ingestion.
sidebar_position: 5
---
UKVisaJobs is the most complex extractor because authenticated sessions are required.
Original website: [my.ukvisajobs.com](https://my.ukvisajobs.com)
## Big picture
Two layers:
1. `extractors/ukvisajobs/src/main.ts` handles login/API calls and dataset output.
2. `orchestrator/src/server/services/ukvisajobs.ts` executes extractor and ingests/de-dupes output.
## 1) Authentication and session cache
Session cache file:
- `extractors/ukvisajobs/storage/ukvisajobs-auth.json`
Flow:
- Reuse cached token/cookies when valid
- Re-login with Playwright + Camoufox when needed
- Refresh and retry on token-expired responses
Force refresh:
- `UKVISAJOBS_REFRESH_ONLY=1`
## 2) API requests
Endpoint:
- `https://my.ukvisajobs.com/ukvisa-api/api/fetch-jobs-data`
Each request includes auth token + session cookies and paginates (15 jobs/page).
## 3) Mapping
- Normalizes salary from min/max/interval
- Builds fallback visa description when content missing
- Maps `job_link` to both `jobUrl` and `applicationLink`
## 4) Output dataset
Written to:
- `extractors/ukvisajobs/storage/datasets/default/`
Includes per-job JSON files and combined `jobs.json`.
## 5) Orchestrator flow
- Spawns extractor (`npx tsx src/main.ts`)
- Runs terms sequentially with delay
- De-dupes by `sourceJobId` (fallback `jobUrl`)
- Fetches detail pages when descriptions are too short
## Controls
- `UKVISAJOBS_EMAIL`, `UKVISAJOBS_PASSWORD`
- `UKVISAJOBS_HEADLESS`
- `UKVISAJOBS_MAX_JOBS` (default 50, max 200)
- `UKVISAJOBS_SEARCH_KEYWORD`
## Practical notes
- Deleting auth cache forces next run to re-login.
- Low-concurrency/polite scraping by design.
- If extractor breaks, check session refresh path first.