Initial setup
This commit is contained in:
DaKheera47
@@ -0,0 +1,46 @@
|
||||
---
|
||||
id: gradcracker
|
||||
title: Gradcracker Extractor
|
||||
description: How the Gradcracker crawler builds search URLs and extracts jobs.
|
||||
sidebar_position: 2
|
||||
---
|
||||
|
||||
A plain-English walkthrough of the Gradcracker extractor in `extractors/gradcracker`.
|
||||
|
||||
## Big picture
|
||||
|
||||
The crawler builds search URLs, scrapes listing pages, then opens job details for descriptions and apply URLs.
|
||||
|
||||
## 1) Build search URLs
|
||||
|
||||
- Combines UK regions with role terms.
|
||||
- Defaults include roles such as `web-development` and `software-systems`.
|
||||
- `GRADCRACKER_SEARCH_TERMS` overrides defaults.
|
||||
|
||||
## 2) Crawl list pages
|
||||
|
||||
- Waits for job cards (`article[wire:key]`).
|
||||
- Extracts title, employer, discipline, deadline, salary, location, degree, start date.
|
||||
- Queues job detail pages.
|
||||
|
||||
Controls:
|
||||
|
||||
- `GRADCRACKER_MAX_JOBS_PER_TERM`
|
||||
- `JOBOPS_SKIP_APPLY_FOR_EXISTING=1`
|
||||
- `JOBOPS_EXISTING_JOB_URLS` / `JOBOPS_EXISTING_JOB_URLS_FILE`
|
||||
|
||||
## 3) Crawl detail pages
|
||||
|
||||
- Waits for `.body-content`
|
||||
- Captures full description text
|
||||
- Clicks apply button to resolve final application URL
|
||||
- Handles popup and same-tab redirects
|
||||
|
||||
## 4) Progress reporting
|
||||
|
||||
Set `JOBOPS_EMIT_PROGRESS=1` for structured progress lines consumable by orchestrator UI.
|
||||
|
||||
## Notes
|
||||
|
||||
- Uses Playwright + Crawlee via Camoufox.
|
||||
- Low concurrency and longer timeouts for stability.
|
||||
@@ -0,0 +1,47 @@
|
||||
---
|
||||
id: jobspy
|
||||
title: JobSpy Extractor
|
||||
description: How the JobSpy Python wrapper is orchestrated and normalized.
|
||||
sidebar_position: 3
|
||||
---
|
||||
|
||||
A walkthrough of the JobSpy extractor for Indeed, LinkedIn, and Glassdoor.
|
||||
|
||||
## Big picture
|
||||
|
||||
JobSpy runs as a Python script per search term, writes JSON, then orchestrator ingests and normalizes into internal job shape.
|
||||
|
||||
## 1) Inputs and defaults
|
||||
|
||||
Key environment variables:
|
||||
|
||||
- `JOBSPY_SITES` (default: `indeed,linkedin`)
|
||||
- `JOBSPY_SEARCH_TERM` (default: `web developer`)
|
||||
- `JOBSPY_LOCATION` (default: `UK`)
|
||||
- `JOBSPY_RESULTS_WANTED` (default: `200`)
|
||||
- `JOBSPY_HOURS_OLD` (default: `72`)
|
||||
- `JOBSPY_COUNTRY_INDEED` (default: `UK`)
|
||||
- `JOBSPY_LINKEDIN_FETCH_DESCRIPTION` (default: `true`)
|
||||
|
||||
## 2) Orchestrator flow
|
||||
|
||||
The service in `orchestrator/src/server/services/jobspy.ts`:
|
||||
|
||||
- Builds search-term list from UI or env
|
||||
- Runs Python once per term with unique output file
|
||||
- Reads JSON and maps to `CreateJobInput`
|
||||
- De-dupes by `jobUrl`
|
||||
- Deletes temp output files best-effort
|
||||
|
||||
## 3) Mapping and cleanup
|
||||
|
||||
- Normalizes salary ranges
|
||||
- Converts empty values to null
|
||||
- Keeps metadata like skills, ratings, remote flags when available
|
||||
- Skips rows with invalid site or missing URL
|
||||
|
||||
## Notes
|
||||
|
||||
- `JOBSPY_SEARCH_TERMS` can be JSON array or `|`, comma, newline-delimited text.
|
||||
- Set `JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0` to speed runs.
|
||||
- Temp output files are stored under `data/imports/`.
|
||||
@@ -0,0 +1,51 @@
|
||||
---
|
||||
id: manual
|
||||
title: Manual Import Extractor
|
||||
description: Import jobs from pasted descriptions and run AI-assisted inference.
|
||||
sidebar_position: 4
|
||||
---
|
||||
|
||||
Manual import lets users add jobs that automated scrapers miss.
|
||||
|
||||
## Big picture
|
||||
|
||||
User pastes raw description, AI infers structure, user reviews edits, then import saves and scores the job.
|
||||
|
||||
## 1) Input
|
||||
|
||||
User pastes a job description in the **Manual Import** UI.
|
||||
|
||||
## 2) AI inference
|
||||
|
||||
Endpoint:
|
||||
|
||||
- `POST /api/manual-jobs/infer`
|
||||
|
||||
Service:
|
||||
|
||||
- `orchestrator/src/server/services/manualJob.ts`
|
||||
|
||||
Behavior:
|
||||
|
||||
- Sends raw text to configured LLM
|
||||
- Extracts structured fields (title, employer, location, salary, etc.)
|
||||
- Returns inferred JSON for user review
|
||||
|
||||
If no LLM key is configured, inference is skipped and user can fill fields manually.
|
||||
|
||||
## 3) Review and edit
|
||||
|
||||
User reviews inferred fields and corrects missing/wrong values.
|
||||
|
||||
## 4) Storage and scoring
|
||||
|
||||
Import endpoint:
|
||||
|
||||
- `POST /api/manual-jobs/import`
|
||||
|
||||
On import:
|
||||
|
||||
- Generates unique job ID if URL absent
|
||||
- Stores source as `manual`
|
||||
- Triggers async suitability scoring
|
||||
- Persists score and reason
|
||||
@@ -0,0 +1,13 @@
|
||||
---
|
||||
id: overview
|
||||
title: Extractors Overview
|
||||
description: Technical index of supported extractors and how they work.
|
||||
sidebar_position: 1
|
||||
---
|
||||
|
||||
Technical breakdowns of each extractor:
|
||||
|
||||
- [Gradcracker](/docs/extractors/gradcracker)
|
||||
- [JobSpy](/docs/extractors/jobspy)
|
||||
- [UKVisaJobs](/docs/extractors/ukvisajobs)
|
||||
- [Manual Import](/docs/extractors/manual)
|
||||
@@ -0,0 +1,73 @@
|
||||
---
|
||||
id: ukvisajobs
|
||||
title: UKVisaJobs Extractor
|
||||
description: Authenticated session flow, API pagination, and orchestrator ingestion.
|
||||
sidebar_position: 5
|
||||
---
|
||||
|
||||
UKVisaJobs is the most complex extractor because authenticated sessions are required.
|
||||
|
||||
## Big picture
|
||||
|
||||
Two layers:
|
||||
|
||||
1. `extractors/ukvisajobs/src/main.ts` handles login/API calls and dataset output.
|
||||
2. `orchestrator/src/server/services/ukvisajobs.ts` executes extractor and ingests/de-dupes output.
|
||||
|
||||
## 1) Authentication and session cache
|
||||
|
||||
Session cache file:
|
||||
|
||||
- `extractors/ukvisajobs/storage/ukvisajobs-auth.json`
|
||||
|
||||
Flow:
|
||||
|
||||
- Reuse cached token/cookies when valid
|
||||
- Re-login with Playwright + Camoufox when needed
|
||||
- Refresh and retry on token-expired responses
|
||||
|
||||
Force refresh:
|
||||
|
||||
- `UKVISAJOBS_REFRESH_ONLY=1`
|
||||
|
||||
## 2) API requests
|
||||
|
||||
Endpoint:
|
||||
|
||||
- `https://my.ukvisajobs.com/ukvisa-api/api/fetch-jobs-data`
|
||||
|
||||
Each request includes auth token + session cookies and paginates (15 jobs/page).
|
||||
|
||||
## 3) Mapping
|
||||
|
||||
- Normalizes salary from min/max/interval
|
||||
- Builds fallback visa description when content missing
|
||||
- Maps `job_link` to both `jobUrl` and `applicationLink`
|
||||
|
||||
## 4) Output dataset
|
||||
|
||||
Written to:
|
||||
|
||||
- `extractors/ukvisajobs/storage/datasets/default/`
|
||||
|
||||
Includes per-job JSON files and combined `jobs.json`.
|
||||
|
||||
## 5) Orchestrator flow
|
||||
|
||||
- Spawns extractor (`npx tsx src/main.ts`)
|
||||
- Runs terms sequentially with delay
|
||||
- De-dupes by `sourceJobId` (fallback `jobUrl`)
|
||||
- Fetches detail pages when descriptions are too short
|
||||
|
||||
## Controls
|
||||
|
||||
- `UKVISAJOBS_EMAIL`, `UKVISAJOBS_PASSWORD`
|
||||
- `UKVISAJOBS_HEADLESS`
|
||||
- `UKVISAJOBS_MAX_JOBS` (default 50, max 200)
|
||||
- `UKVISAJOBS_SEARCH_KEYWORD`
|
||||
|
||||
## Practical notes
|
||||
|
||||
- Deleting auth cache forces next run to re-login.
|
||||
- Low-concurrency/polite scraping by design.
|
||||
- If extractor breaks, check session refresh path first.
|
||||
Reference in New Issue
Block a user