---
id: jobspy
title: JobSpy Extractor
description: How the JobSpy Python wrapper is orchestrated and normalized.
sidebar_position: 3
---

A walkthrough of the JobSpy extractor for Indeed, LinkedIn, and Glassdoor.

Original websites:
- [indeed.com](https://www.indeed.com)
- [linkedin.com/jobs](https://www.linkedin.com/jobs)
- [glassdoor.com](https://www.glassdoor.com)

## Big picture

JobSpy runs as a Python script per search term, writes JSON, then orchestrator ingests and normalizes into internal job shape.

## 1) Inputs and defaults

Key environment variables:

- `JOBSPY_SITES` (default: `indeed,linkedin`)
- `JOBSPY_SEARCH_TERM` (default: `web developer`)
- `JOBSPY_LOCATION` (default: `UK`)
- `JOBSPY_RESULTS_WANTED` (default: `200`)
- `JOBSPY_HOURS_OLD` (default: `72`)
- `JOBSPY_COUNTRY_INDEED` (default: `UK`)
- `JOBSPY_LINKEDIN_FETCH_DESCRIPTION` (default: `true`)
- `JOBSPY_IS_REMOTE` (unset by default)

## 2) Orchestrator flow

The service in `orchestrator/src/server/services/jobspy.ts`:

- Builds search-term list from UI or env
- Runs Python once per term with unique output file
- Reads JSON and maps to `CreateJobInput`
- De-dupes by `jobUrl`
- Deletes temp output files best-effort

## 3) Mapping and cleanup

- Normalizes salary ranges
- Converts empty values to null
- Keeps metadata like skills, ratings, remote flags when available
- Skips rows with invalid site or missing URL

## Notes

- `JOBSPY_SEARCH_TERMS` can be JSON array or `|`, comma, newline-delimited text.
- Set `JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0` to speed runs.
- Temp output files are stored under `data/imports/`.
- If workplace type is only `Remote`, JobSpy runs with `JOBSPY_IS_REMOTE=true`.
- If workplace type includes `Hybrid` or `Onsite`, JobSpy cannot enforce those filters precisely, so the JobSpy-backed sources run without a workplace-type filter and may return broader results.

## Common Problems

- `Hybrid` or `Onsite` was selected, but Indeed, LinkedIn, or Glassdoor still returned remote jobs.
  JobSpy only supports a strict remote toggle. Any workplace-type selection that includes `Hybrid` or `Onsite` broadens those source results.
- A run returned fewer LinkedIn descriptions than expected.
  `JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0` disables description fetching to speed up runs.
- Different cities need different workplace-type filters.
  This is not supported in the current automatic-run flow. JobSpy receives one global workplace-type selection per run/query invocation.
- Glassdoor always returns 0 jobs (fixed).
  Upstream `python-jobspy` (through at least `1.1.82`) discards an entire page of Glassdoor results whenever Glassdoor's GraphQL response includes *any* `errors` entry — even for an unrelated field. In practice, Glassdoor's backend consistently returns a partial error for `jobsPageSeoData` (`SUBREQUEST_HTTP_ERROR` / "dns error" on Glassdoor's own infra) while the actual job listings still come back fine, so every Glassdoor run silently returned 0 jobs. `extractors/jobspy/scrape_jobs.py` monkeypatches `Glassdoor._fetch_jobs_page` at import time to only treat the response as failed when `data.jobListings` is actually missing, and logs a warning (not an error) when it ignores an unrelated partial error. `requirements.txt` pins `python-jobspy==1.1.82` so an upstream version bump doesn't silently drop this patch — re-verify the patched method still matches before removing the pin.
- A Glassdoor run logs `429` / "Blocked by Glassdoor for too many requests".
  This is a real rate limit from running too many Glassdoor requests back-to-back (e.g. repeated manual test runs). Space out runs; the automatic pipeline's normal cadence does not trigger this in practice.