Jobber/docs-site/docs/extractors/jobspy.md

id, title, description, sidebar_position

id	title	description	sidebar_position
jobspy	JobSpy Extractor	How the JobSpy Python wrapper is orchestrated and normalized.	3

A walkthrough of the JobSpy extractor for Indeed, LinkedIn, and Glassdoor.

Original websites:

• indeed.com
• linkedin.com/jobs
• glassdoor.com

Big picture

JobSpy runs as a Python script per search term, writes JSON, then orchestrator ingests and normalizes into internal job shape.

1) Inputs and defaults

Key environment variables:

• JOBSPY_SITES (default: indeed,linkedin)
• JOBSPY_SEARCH_TERM (default: web developer)
• JOBSPY_LOCATION (default: UK)
• JOBSPY_RESULTS_WANTED (default: 200)
• JOBSPY_HOURS_OLD (default: 72)
• JOBSPY_COUNTRY_INDEED (default: UK)
• JOBSPY_LINKEDIN_FETCH_DESCRIPTION (default: true)
• JOBSPY_IS_REMOTE (unset by default)

2) Orchestrator flow

The service in orchestrator/src/server/services/jobspy.ts:

• Builds search-term list from UI or env
• Runs Python once per term with unique output file
• Reads JSON and maps to CreateJobInput
• De-dupes by jobUrl
• Deletes temp output files best-effort

3) Mapping and cleanup

• Normalizes salary ranges
• Converts empty values to null
• Keeps metadata like skills, ratings, remote flags when available
• Skips rows with invalid site or missing URL

Notes

• JOBSPY_SEARCH_TERMS can be JSON array or |, comma, newline-delimited text.
• Set JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0 to speed runs.
• Temp output files are stored under data/imports/.
• If workplace type is only Remote, JobSpy runs with JOBSPY_IS_REMOTE=true.
• If workplace type includes Hybrid or Onsite, JobSpy cannot enforce those filters precisely, so the JobSpy-backed sources run without a workplace-type filter and may return broader results.

Common Problems

• Hybrid or Onsite was selected, but Indeed, LinkedIn, or Glassdoor still returned remote jobs. JobSpy only supports a strict remote toggle. Any workplace-type selection that includes Hybrid or Onsite broadens those source results.
• A run returned fewer LinkedIn descriptions than expected. JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0 disables description fetching to speed up runs.
• Different cities need different workplace-type filters. This is not supported in the current automatic-run flow. JobSpy receives one global workplace-type selection per run/query invocation.
• Glassdoor always returns 0 jobs (fixed). Upstream python-jobspy (through at least 1.1.82) discards an entire page of Glassdoor results whenever Glassdoor's GraphQL response includes any errors entry — even for an unrelated field. In practice, Glassdoor's backend consistently returns a partial error for jobsPageSeoData (SUBREQUEST_HTTP_ERROR / "dns error" on Glassdoor's own infra) while the actual job listings still come back fine, so every Glassdoor run silently returned 0 jobs. extractors/jobspy/scrape_jobs.py monkeypatches Glassdoor._fetch_jobs_page at import time to only treat the response as failed when data.jobListings is actually missing, and logs a warning (not an error) when it ignores an unrelated partial error. requirements.txt pins python-jobspy==1.1.82 so an upstream version bump doesn't silently drop this patch — re-verify the patched method still matches before removing the pin.
• A Glassdoor run logs 429 / "Blocked by Glassdoor for too many requests". This is a real rate limit from running too many Glassdoor requests back-to-back (e.g. repeated manual test runs). Space out runs; the automatic pipeline's normal cadence does not trigger this in practice.