feat: add Google Jobs extractor and fix scraper reliability gaps
Ship a Camoufox-backed Google Jobs source, restore Glassdoor results discarded by a python-jobspy GraphQL quirk, and fix Himalayas/Gradcracker zero-job failures. Also harden prior-skip dismiss matching and multi-profile basic-auth switching for CA/US runs.
This commit is contained in:
+29
-1
@@ -260,7 +260,35 @@ ADZUNA_APP_KEY=
|
||||
# =============================================================================
|
||||
# Caps via Settings: workingnomadsMaxJobsPerTerm, testdevjobsMaxJobsPerTerm,
|
||||
# testdevjobsMaxPages, builtinMaxJobsPerTerm, builtinMaxPagesPerTerm,
|
||||
# wellfoundMaxJobsPerTerm (Wellfound uses Camoufox when installed).
|
||||
# wellfoundMaxJobsPerTerm, googleJobsMaxJobsPerTerm (Wellfound and Google
|
||||
# Jobs both use Camoufox when installed).
|
||||
|
||||
# =============================================================================
|
||||
# Google Jobs — optional (auto-enabled, no credentials, unofficial scrape)
|
||||
# =============================================================================
|
||||
# GOOGLE_JOBS_PROXY=http://user:pass@proxy-host:port
|
||||
# ^ Most reliable way to avoid Google's CAPTCHA/rate-limit wall (residential or
|
||||
# mobile proxy recommended; datacenter IPs, including most VPS/cloud hosts,
|
||||
# are aggressively blocked). Applies to both manual runs and the pipeline.
|
||||
#
|
||||
# GOOGLE_JOBS_HEADLESS=false
|
||||
# ^ Manual/local use ONLY (do not set this in the server .env for the
|
||||
# unattended pipeline/cron — a headless server has no display for a human
|
||||
# to see the browser). Opens a real Firefox window so you can solve a
|
||||
# CAPTCHA by hand; waits up to GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS (default
|
||||
# 300000 = 5 min) before giving up. Run directly, e.g.:
|
||||
# GOOGLE_JOBS_HEADLESS=false npm --workspace google-jobs-extractor run start
|
||||
# Session cookies are saved to extractors/google-jobs/storage/state.json
|
||||
# and reused by later runs (including headless/automated ones), which can
|
||||
# reduce how often future runs hit a CAPTCHA.
|
||||
#
|
||||
# Caps via Settings: googleJobsMaxJobsPerTerm (default 30, max 150).
|
||||
#
|
||||
# Note: there is no country/location override for this extractor — Google
|
||||
# resolves location from the browser's real (or GOOGLE_JOBS_PROXY's) IP.
|
||||
# Forcing a mismatched country has been observed to make Google's Jobs
|
||||
# vertical return "no matches" even though the tab loads correctly. To
|
||||
# target a different country's jobs, use a proxy with an exit node there.
|
||||
|
||||
# =============================================================================
|
||||
# Teamtailor / Huntflow / Factorial / Career pages — optional
|
||||
|
||||
Reference in New Issue
Block a user