Jobber/docs-site/docs/extractors/google-jobs.md

id, title, description, sidebar_position

id	title	description	sidebar_position
google-jobs	Google Jobs Extractor	Browser-backed scraping of the Google for Jobs search widget.	8

What it is

Original feature: Google for Jobs

Google Jobs is a browser-backed extractor that opens Google's Jobs vertical (udm=8) in Firefox (Camoufox when available) for each pipeline search term, scrolls the results panel to load additional cards, then clicks into each card to resolve a real "Apply on <site>" link and description before mapping rows into the orchestrator CreateJobInput shape.

Implementation split:

1. extractors/google-jobs/src/main.ts builds the search URL, waits for the jobs list, scrolls to lazy-load more cards, extracts card metadata (title/employer/location/"via" source), clicks each card to read the detail pane, and writes dataset JSON.
2. extractors/google-jobs/src/run.ts spawns the browser subprocess, applies a hard timeout, reads the dataset, and maps rows to CreateJobInput.
3. extractors/google-jobs/manifest.ts wires pipeline searchTerms into the extractor.

A note on the search URL and location

Google's Jobs vertical is reached with udm=8 (its current internal vertical id). An older, commonly-documented deep link — ibp=htl;jobs — still works but gets silently redirected to udm=8 anyway, and combining it with extra parameters (a num= result-count hint, or a gl= country override that doesn't match the browser's real/proxied IP) has been observed to make Google return "no matches" or "can't determine location" even though the tab itself loads correctly.

Because of that, this extractor does not send a gl (country) parameter — location is resolved by Google from the browser's real IP (or GOOGLE_JOBS_PROXY's exit IP), which Camoufox's geoip option keeps internally consistent. To target a different country's jobs, use a residential proxy with an exit node in that country rather than trying to override the country via a parameter.

Dealing with Google's CAPTCHA / bot detection

Google actively fingerprints and challenges automated browser traffic, especially from datacenter/VPS IPs (which most servers running JobOps' cron pipeline will have). The extractor has layered mitigations, from always-on to opt-in:

1. Always on: Camoufox launches with humanize: true and geoip: true (human-like cursor movement, IP-consistent locale/geolocation) to reduce — not eliminate — fingerprint-based detection.
2. Always on: browser cookies/session are persisted to extractors/google-jobs/storage/state.json and reused on the next run. A session that has cleared a CAPTCHA once (including via option 4 below) tends to see fewer challenges for a while afterward.
3. Opt-in, most reliable: set GOOGLE_JOBS_PROXY (e.g. a residential/mobile proxy URL) to avoid datacenter-IP blocks in the first place. This is the standard real-world fix; unproxied runs from cloud servers should expect to be blocked often.
4. Opt-in, manual/local only: set GOOGLE_JOBS_HEADLESS=false and run the extractor directly (not via the pipeline) to open a real, visible Firefox window. If Google shows a CAPTCHA, solve it yourself in that window — the script polls and automatically resumes once it clears (up to GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS, default 5 minutes). The resulting session cookies are saved and reused by later automated/headless runs. Do not set GOOGLE_JOBS_HEADLESS=false in the server's always-on .env — a headless server has no display for a human to interact with, and the run will just wait until the timeout.

Why it exists

Google for Jobs aggregates postings across the open web — including boards and employer career pages that JobOps does not otherwise scrape. Adding it as a proper source closes some of the gap between "jobs found by manually Googling a role" and "jobs discovered by the pipeline."

It is not a general web crawler and does not replace dedicated extractors: it only surfaces whatever Google's Jobs widget chooses to show for a query, subject to Google's own ranking, freshness, and regional availability.

How to use it

1. Open Run jobs and choose Automatic.
2. Google Jobs is enabled by default in Sources (toggle it off if you do not want it for this run).
3. Set your existing automatic run knobs:
   • searchTerms become the "<term> jobs" query sent to Google per term.
   • googleJobsMaxJobsPerTerm (Settings) caps jobs scraped per run (default 30, max 150).
   • Location is derived from the browser's real/proxied IP, not from a country setting — see the note above.
4. Start the run and watch progress in the pipeline progress card.

Local run example (headless, no proxy):

GOOGLE_JOBS_SEARCH_TERMS='["automation engineer"]' \
GOOGLE_JOBS_MAX_JOBS='10' \
npm --workspace google-jobs-extractor run start

Local run example (headed, solve the CAPTCHA yourself if one appears):

GOOGLE_JOBS_HEADLESS=false \
GOOGLE_JOBS_SEARCH_TERMS='["automation engineer"]' \
npm --workspace google-jobs-extractor run start

Defaults and constraints:

• No credentials required; this is a direct HTML/DOM scrape of Google's public search results, not an official API (Google does not offer a free Google Jobs API).
• Google's markup and CSS class names change frequently and are obfuscated; the scraper favors known current card classes (e.g. .EimVGf) with multiple fallback selectors and a leaf-node text heuristic, but selectors can still rot and return zero results until updated.
• Every card carries Google's own data-share-url permalink for that specific listing, which the extractor uses as jobUrl — so a job stays unique and reviewable even if the "click into the card for a detail pane" step below fails for it.
• Because resolving a real application link requires clicking each card in a real browser, this extractor is slower per job than API-backed sources — keep googleJobsMaxJobsPerTerm modest for frequent runs.
• If no external "Apply on <site>" link is found for a card, the extractor falls back to Google's own permalink for that listing so it's still reviewable, but not directly one-click-applyable.
• After scraping, JobOps probes each apply URL (preferring the external link over Google's share permalink). Listings that return 404/410 or pages that say the job expired / is no longer available are dropped before import. Network failures fail-open (job is kept).
• Automated, frequent, or high-volume scraping of Google search results may be against Google's Terms of Service; treat this extractor as best-effort and keep run frequency/volume conservative.
• See Dealing with Google's CAPTCHA / bot detection above for proxy and interactive-solve options.

Common problems

Google Jobs returns 0 jobs for a term

The extractor logs one of two distinct messages to help tell these apart:

• Google's Jobs tab reported no matches for "<term>" — Google correctly landed on the Jobs tab but genuinely has no results for that query/locale. Try a broader, more common phrasing (e.g. "automation engineer" instead of a very narrow title).
• Jobs list did not appear for "<term>" — an unrecognized page state (consent wall, layout change, or rotted selectors). The extractor saves a screenshot and full HTML dump to extractors/google-jobs/storage/debug-last.png / debug-last.html for inspection; compare it against a fresh manual search to see what changed.

Results look sparse or missing company/location

• The list-view heuristics rely on leaf DOM nodes near the job title; if Google restructures the card markup, company/location extraction can degrade even when titles still work. This is expected best-effort behavior for an unofficial scrape.

Run is slow

• Each job requires a real click + detail-pane render, unlike single-request API extractors. Lower googleJobsMaxJobsPerTerm or reduce the number of search terms for faster runs.

Blocked, CAPTCHA'd, or consistently empty results

• Google may rate-limit or challenge automated browser traffic, especially from datacenter/VPS IPs. In order of effort:
  1. Set GOOGLE_JOBS_PROXY to a residential/mobile proxy — the most reliable fix for unattended/server runs.
  2. Run locally with GOOGLE_JOBS_HEADLESS=false and solve the CAPTCHA yourself once; the saved session (storage/state.json) may reduce blocks on subsequent headless runs for a while.
  3. Reduce run frequency, lower the per-term cap, or disable the source for that run and rely on other extractors / Manual Import instead.

Related pages

• Extractors Overview
• Manual Import
• Pipeline Run
• Settings