feat: add Google Jobs extractor and fix scraper reliability gaps

Ship a Camoufox-backed Google Jobs source, restore Glassdoor results discarded by a python-jobspy GraphQL quirk, and fix Himalayas/Gradcracker zero-job failures. Also harden prior-skip dismiss matching and multi-profile basic-auth switching for CA/US runs.
2026-07-09 15:11:53 -04:00
parent 84e6835b11
commit 40c7cdece3
47 changed files with 1989 additions and 87 deletions
@@ -61,4 +61,8 @@ The service in `orchestrator/src/server/services/jobspy.ts`:
 - A run returned fewer LinkedIn descriptions than expected.
  `JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0` disables description fetching to speed up runs.
 - Different cities need different workplace-type filters.
-  This is not supported in the current automatic-run flow. JobSpy receives one global workplace-type selection per run/query invocation.
+  This is not supported in the current automatic-run flow. JobSpy receives one global workplace-type selection per run/query invocation.
+- Glassdoor always returns 0 jobs (fixed).
+  Upstream `python-jobspy` (through at least `1.1.82`) discards an entire page of Glassdoor results whenever Glassdoor's GraphQL response includes *any* `errors` entry — even for an unrelated field. In practice, Glassdoor's backend consistently returns a partial error for `jobsPageSeoData` (`SUBREQUEST_HTTP_ERROR` / "dns error" on Glassdoor's own infra) while the actual job listings still come back fine, so every Glassdoor run silently returned 0 jobs. `extractors/jobspy/scrape_jobs.py` monkeypatches `Glassdoor._fetch_jobs_page` at import time to only treat the response as failed when `data.jobListings` is actually missing, and logs a warning (not an error) when it ignores an unrelated partial error. `requirements.txt` pins `python-jobspy==1.1.82` so an upstream version bump doesn't silently drop this patch — re-verify the patched method still matches before removing the pin.
+- A Glassdoor run logs `429` / "Blocked by Glassdoor for too many requests".
+  This is a real rate limit from running too many Glassdoor requests back-to-back (e.g. repeated manual test runs). Space out runs; the automatic pipeline's normal cadence does not trigger this in practice.