feat: add Google Jobs extractor and fix scraper reliability gaps
CI / skip-ci-check (pull_request) Successful in 8s
CI / docker-ci (pull_request) Successful in 10s
CI / secret-scan (pull_request) Successful in 7s

Ship a Camoufox-backed Google Jobs source, restore Glassdoor results discarded by a python-jobspy GraphQL quirk, and fix Himalayas/Gradcracker zero-job failures. Also harden prior-skip dismiss matching and multi-profile basic-auth switching for CA/US runs.
This commit is contained in:
2026-07-09 15:11:53 -04:00
parent 84e6835b11
commit 40c7cdece3
47 changed files with 1989 additions and 87 deletions
+29 -1
View File
@@ -260,7 +260,35 @@ ADZUNA_APP_KEY=
# =============================================================================
# Caps via Settings: workingnomadsMaxJobsPerTerm, testdevjobsMaxJobsPerTerm,
# testdevjobsMaxPages, builtinMaxJobsPerTerm, builtinMaxPagesPerTerm,
# wellfoundMaxJobsPerTerm (Wellfound uses Camoufox when installed).
# wellfoundMaxJobsPerTerm, googleJobsMaxJobsPerTerm (Wellfound and Google
# Jobs both use Camoufox when installed).
# =============================================================================
# Google Jobs — optional (auto-enabled, no credentials, unofficial scrape)
# =============================================================================
# GOOGLE_JOBS_PROXY=http://user:pass@proxy-host:port
# ^ Most reliable way to avoid Google's CAPTCHA/rate-limit wall (residential or
# mobile proxy recommended; datacenter IPs, including most VPS/cloud hosts,
# are aggressively blocked). Applies to both manual runs and the pipeline.
#
# GOOGLE_JOBS_HEADLESS=false
# ^ Manual/local use ONLY (do not set this in the server .env for the
# unattended pipeline/cron — a headless server has no display for a human
# to see the browser). Opens a real Firefox window so you can solve a
# CAPTCHA by hand; waits up to GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS (default
# 300000 = 5 min) before giving up. Run directly, e.g.:
# GOOGLE_JOBS_HEADLESS=false npm --workspace google-jobs-extractor run start
# Session cookies are saved to extractors/google-jobs/storage/state.json
# and reused by later runs (including headless/automated ones), which can
# reduce how often future runs hit a CAPTCHA.
#
# Caps via Settings: googleJobsMaxJobsPerTerm (default 30, max 150).
#
# Note: there is no country/location override for this extractor — Google
# resolves location from the browser's real (or GOOGLE_JOBS_PROXY's) IP.
# Forcing a mismatched country has been observed to make Google's Jobs
# vertical return "no matches" even though the tab loads correctly. To
# target a different country's jobs, use a proxy with an exit node there.
# =============================================================================
# Teamtailor / Huntflow / Factorial / Career pages — optional