feat: add Google Jobs extractor and fix scraper reliability gaps
Ship a Camoufox-backed Google Jobs source, restore Glassdoor results discarded by a python-jobspy GraphQL quirk, and fix Himalayas/Gradcracker zero-job failures. Also harden prior-skip dismiss matching and multi-profile basic-auth switching for CA/US runs.
This commit is contained in:
@@ -32,10 +32,11 @@ Eluta surfaces Canadian roles indexed directly from employer career sites, often
|
||||
### Empty feeds
|
||||
|
||||
- The `location` string may be too broad or spelled differently than Eluta expects. Try a major city plus province (e.g. `Calgary, AB`).
|
||||
- Eluta's RSS is a small recent slice of *all* jobs for that location (often retail/hospitality-heavy). Pipeline search terms like `software engineer` can legitimately match zero items even when the feed itself is healthy. Broaden terms or add more metro locations.
|
||||
|
||||
### RSS HTTP errors
|
||||
### RSS HTTP errors / "Too Many Requests"
|
||||
|
||||
- Eluta may block unusual clients; the extractor sends a conventional User-Agent. Retry later or reduce the number of location feeds per run.
|
||||
- Eluta rate-limits aggressive or browser-like clients. The extractor uses a dedicated `JobOps/... Eluta RSS consumer` User-Agent; retry later or reduce the number of location feeds per run.
|
||||
|
||||
## Related pages
|
||||
|
||||
|
||||
@@ -0,0 +1,106 @@
|
||||
---
|
||||
id: google-jobs
|
||||
title: Google Jobs Extractor
|
||||
description: Browser-backed scraping of the Google for Jobs search widget.
|
||||
sidebar_position: 8
|
||||
---
|
||||
|
||||
## What it is
|
||||
|
||||
Original feature: [Google for Jobs](https://www.google.com/search?q=jobs&udm=8)
|
||||
|
||||
Google Jobs is a browser-backed extractor that opens Google's Jobs vertical (`udm=8`) in Firefox (Camoufox when available) for each pipeline search term, scrolls the results panel to load additional cards, then clicks into each card to resolve a real "Apply on <site>" link and description before mapping rows into the orchestrator `CreateJobInput` shape.
|
||||
|
||||
Implementation split:
|
||||
|
||||
1. `extractors/google-jobs/src/main.ts` builds the search URL, waits for the jobs list, scrolls to lazy-load more cards, extracts card metadata (title/employer/location/"via" source), clicks each card to read the detail pane, and writes dataset JSON.
|
||||
2. `extractors/google-jobs/src/run.ts` spawns the browser subprocess, applies a hard timeout, reads the dataset, and maps rows to `CreateJobInput`.
|
||||
3. `extractors/google-jobs/manifest.ts` wires pipeline `searchTerms` into the extractor.
|
||||
|
||||
### A note on the search URL and location
|
||||
|
||||
Google's Jobs vertical is reached with `udm=8` (its current internal vertical id). An older, commonly-documented deep link — `ibp=htl;jobs` — still works but gets silently redirected to `udm=8` anyway, and combining it with extra parameters (a `num=` result-count hint, or a `gl=` country override that doesn't match the browser's real/proxied IP) has been observed to make Google return "no matches" or "can't determine location" even though the tab itself loads correctly.
|
||||
|
||||
Because of that, this extractor does **not** send a `gl` (country) parameter — location is resolved by Google from the browser's real IP (or `GOOGLE_JOBS_PROXY`'s exit IP), which Camoufox's `geoip` option keeps internally consistent. To target a different country's jobs, use a residential proxy with an exit node in that country rather than trying to override the country via a parameter.
|
||||
|
||||
### Dealing with Google's CAPTCHA / bot detection
|
||||
|
||||
Google actively fingerprints and challenges automated browser traffic, especially from datacenter/VPS IPs (which most servers running JobOps' cron pipeline will have). The extractor has layered mitigations, from always-on to opt-in:
|
||||
|
||||
1. **Always on:** Camoufox launches with `humanize: true` and `geoip: true` (human-like cursor movement, IP-consistent locale/geolocation) to reduce — not eliminate — fingerprint-based detection.
|
||||
2. **Always on:** browser cookies/session are persisted to `extractors/google-jobs/storage/state.json` and reused on the next run. A session that has cleared a CAPTCHA once (including via option 4 below) tends to see fewer challenges for a while afterward.
|
||||
3. **Opt-in, most reliable:** set `GOOGLE_JOBS_PROXY` (e.g. a residential/mobile proxy URL) to avoid datacenter-IP blocks in the first place. This is the standard real-world fix; unproxied runs from cloud servers should expect to be blocked often.
|
||||
4. **Opt-in, manual/local only:** set `GOOGLE_JOBS_HEADLESS=false` and run the extractor directly (not via the pipeline) to open a real, visible Firefox window. If Google shows a CAPTCHA, solve it yourself in that window — the script polls and automatically resumes once it clears (up to `GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS`, default 5 minutes). The resulting session cookies are saved and reused by later automated/headless runs. **Do not** set `GOOGLE_JOBS_HEADLESS=false` in the server's always-on `.env` — a headless server has no display for a human to interact with, and the run will just wait until the timeout.
|
||||
|
||||
## Why it exists
|
||||
|
||||
Google for Jobs aggregates postings across the open web — including boards and employer career pages that JobOps does not otherwise scrape. Adding it as a proper source closes some of the gap between "jobs found by manually Googling a role" and "jobs discovered by the pipeline."
|
||||
|
||||
It is **not** a general web crawler and does not replace dedicated extractors: it only surfaces whatever Google's Jobs widget chooses to show for a query, subject to Google's own ranking, freshness, and regional availability.
|
||||
|
||||
## How to use it
|
||||
|
||||
1. Open **Run jobs** and choose **Automatic**.
|
||||
2. **Google Jobs** is enabled by default in **Sources** (toggle it off if you do not want it for this run).
|
||||
3. Set your existing automatic run knobs:
|
||||
- `searchTerms` become the `"<term> jobs"` query sent to Google per term.
|
||||
- `googleJobsMaxJobsPerTerm` (Settings) caps jobs scraped per run (default `30`, max `150`).
|
||||
- Location is derived from the browser's real/proxied IP, not from a country setting — see [the note above](#a-note-on-the-search-url-and-location).
|
||||
4. Start the run and watch progress in the pipeline progress card.
|
||||
|
||||
Local run example (headless, no proxy):
|
||||
|
||||
```bash
|
||||
GOOGLE_JOBS_SEARCH_TERMS='["automation engineer"]' \
|
||||
GOOGLE_JOBS_MAX_JOBS='10' \
|
||||
npm --workspace google-jobs-extractor run start
|
||||
```
|
||||
|
||||
Local run example (headed, solve the CAPTCHA yourself if one appears):
|
||||
|
||||
```bash
|
||||
GOOGLE_JOBS_HEADLESS=false \
|
||||
GOOGLE_JOBS_SEARCH_TERMS='["automation engineer"]' \
|
||||
npm --workspace google-jobs-extractor run start
|
||||
```
|
||||
|
||||
Defaults and constraints:
|
||||
|
||||
- No credentials required; this is a direct HTML/DOM scrape of Google's public search results, not an official API (Google does not offer a free Google Jobs API).
|
||||
- Google's markup and CSS class names change frequently and are obfuscated; the scraper favors known current card classes (e.g. `.EimVGf`) with multiple fallback selectors and a leaf-node text heuristic, but selectors can still rot and return zero results until updated.
|
||||
- Every card carries Google's own `data-share-url` permalink for that specific listing, which the extractor uses as `jobUrl` — so a job stays unique and reviewable even if the "click into the card for a detail pane" step below fails for it.
|
||||
- Because resolving a real application link requires clicking each card in a real browser, this extractor is slower per job than API-backed sources — keep `googleJobsMaxJobsPerTerm` modest for frequent runs.
|
||||
- If no external "Apply on <site>" link is found for a card, the extractor falls back to Google's own permalink for that listing so it's still reviewable, but not directly one-click-applyable.
|
||||
- Automated, frequent, or high-volume scraping of Google search results may be against Google's Terms of Service; treat this extractor as best-effort and keep run frequency/volume conservative.
|
||||
- See [Dealing with Google's CAPTCHA / bot detection](#dealing-with-googles-captcha--bot-detection) above for proxy and interactive-solve options.
|
||||
|
||||
## Common problems
|
||||
|
||||
### Google Jobs returns 0 jobs for a term
|
||||
|
||||
The extractor logs one of two distinct messages to help tell these apart:
|
||||
|
||||
- `Google's Jobs tab reported no matches for "<term>"` — Google correctly landed on the Jobs tab but genuinely has no results for that query/locale. Try a broader, more common phrasing (e.g. `"automation engineer"` instead of a very narrow title).
|
||||
- `Jobs list did not appear for "<term>"` — an unrecognized page state (consent wall, layout change, or rotted selectors). The extractor saves a screenshot and full HTML dump to `extractors/google-jobs/storage/debug-last.png` / `debug-last.html` for inspection; compare it against a fresh manual search to see what changed.
|
||||
|
||||
### Results look sparse or missing company/location
|
||||
|
||||
- The list-view heuristics rely on leaf DOM nodes near the job title; if Google restructures the card markup, company/location extraction can degrade even when titles still work. This is expected best-effort behavior for an unofficial scrape.
|
||||
|
||||
### Run is slow
|
||||
|
||||
- Each job requires a real click + detail-pane render, unlike single-request API extractors. Lower `googleJobsMaxJobsPerTerm` or reduce the number of search terms for faster runs.
|
||||
|
||||
### Blocked, CAPTCHA'd, or consistently empty results
|
||||
|
||||
- Google may rate-limit or challenge automated browser traffic, especially from datacenter/VPS IPs. In order of effort:
|
||||
1. Set `GOOGLE_JOBS_PROXY` to a residential/mobile proxy — the most reliable fix for unattended/server runs.
|
||||
2. Run locally with `GOOGLE_JOBS_HEADLESS=false` and solve the CAPTCHA yourself once; the saved session (`storage/state.json`) may reduce blocks on subsequent headless runs for a while.
|
||||
3. Reduce run frequency, lower the per-term cap, or disable the source for that run and rely on other extractors / Manual Import instead.
|
||||
|
||||
## Related pages
|
||||
|
||||
- [Extractors Overview](/docs/next/extractors/overview)
|
||||
- [Manual Import](/docs/next/extractors/manual)
|
||||
- [Pipeline Run](/docs/next/features/pipeline-run)
|
||||
- [Settings](/docs/next/features/settings)
|
||||
@@ -46,3 +46,11 @@ Set `JOBOPS_EMIT_PROGRESS=1` for structured progress lines consumable by orchest
|
||||
|
||||
- Uses Playwright + Crawlee via Camoufox.
|
||||
- Low concurrency and longer timeouts for stability.
|
||||
- Employer logo (`figure img` alt text) and employer link are optional. Cards that omit them used to throw a Playwright `getAttribute` timeout and fail the whole region page (0 jobs saved even after cards were counted). Missing employer fields are now skipped per card.
|
||||
|
||||
## Common problems
|
||||
|
||||
### Region pages time out / run returns 0 jobs
|
||||
|
||||
- Gradcracker Livewire markup sometimes omits the employer logo on a card. Older builds aborted the whole page on that timeout. Current builds continue and leave employer blank.
|
||||
- Anti-bot / Camoufox issues: ensure Camoufox is installed (`npx camoufox-js fetch`) and prefer headed runs when debugging.
|
||||
|
||||
@@ -0,0 +1,45 @@
|
||||
---
|
||||
id: himalayas
|
||||
title: Himalayas Extractor
|
||||
description: Public remote-jobs API pagination and client-side term filtering.
|
||||
sidebar_position: 16
|
||||
---
|
||||
|
||||
## What it is
|
||||
|
||||
Original site: [himalayas.app](https://himalayas.app)
|
||||
|
||||
The extractor lives in `extractors/himalayas/manifest.ts`. It paginates the public JSON API (`https://himalayas.app/jobs/api?limit=&offset=`), filters rows client-side by pipeline search terms (title + categories), and maps matches into `CreateJobInput`.
|
||||
|
||||
## Why it exists
|
||||
|
||||
Himalayas is a large remote-jobs index with a stable, unauthenticated API — useful for remote-first discovery without browser automation or credentials.
|
||||
|
||||
## How to use it
|
||||
|
||||
1. Enable **Himalayas** in pipeline sources.
|
||||
2. Optionally set **Himalayas max jobs per term** (`himalayasMaxJobsPerTerm`, default `100`).
|
||||
3. Pipeline `searchTerms` filter titles/categories client-side (substring match, case-insensitive).
|
||||
4. Run the pipeline.
|
||||
|
||||
### Defaults and constraints
|
||||
|
||||
- The upstream API **silently caps `limit` at 20** per request. The extractor uses `PAGE_SIZE = 20` and paginates with `offset` (up to 10 pages / 200 rows scanned per run before the per-term cap).
|
||||
- There is no server-side search — if a term is rare, early pages may not contain matches even though later pages do. Raise the page budget only by changing the extractor constants if you need deeper scans.
|
||||
- No auth required.
|
||||
|
||||
## Common problems
|
||||
|
||||
### Zero jobs for a term I know exists
|
||||
|
||||
- The term may not appear in the first ~200 API rows (newest-first feed). Try a shorter token (`software`, `engineer`, `SDET`) or accept that Himalayas skews toward whatever is currently at the top of the feed.
|
||||
- Confirm you are not hitting an old build that requested `limit=50` and stopped after one page (fixed: the API returns at most 20, which used to look like end-of-feed).
|
||||
|
||||
### Rate limits / HTTP errors
|
||||
|
||||
- Retry later; the public API is generally stable but can throttle burst traffic.
|
||||
|
||||
## Related pages
|
||||
|
||||
- [Extractors Overview](/docs/next/extractors/overview)
|
||||
- [Add an Extractor](/docs/next/workflows/add-an-extractor)
|
||||
@@ -61,4 +61,8 @@ The service in `orchestrator/src/server/services/jobspy.ts`:
|
||||
- A run returned fewer LinkedIn descriptions than expected.
|
||||
`JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0` disables description fetching to speed up runs.
|
||||
- Different cities need different workplace-type filters.
|
||||
This is not supported in the current automatic-run flow. JobSpy receives one global workplace-type selection per run/query invocation.
|
||||
This is not supported in the current automatic-run flow. JobSpy receives one global workplace-type selection per run/query invocation.
|
||||
- Glassdoor always returns 0 jobs (fixed).
|
||||
Upstream `python-jobspy` (through at least `1.1.82`) discards an entire page of Glassdoor results whenever Glassdoor's GraphQL response includes *any* `errors` entry — even for an unrelated field. In practice, Glassdoor's backend consistently returns a partial error for `jobsPageSeoData` (`SUBREQUEST_HTTP_ERROR` / "dns error" on Glassdoor's own infra) while the actual job listings still come back fine, so every Glassdoor run silently returned 0 jobs. `extractors/jobspy/scrape_jobs.py` monkeypatches `Glassdoor._fetch_jobs_page` at import time to only treat the response as failed when `data.jobListings` is actually missing, and logs a warning (not an error) when it ignores an unrelated partial error. `requirements.txt` pins `python-jobspy==1.1.82` so an upstream version bump doesn't silently drop this patch — re-verify the patched method still matches before removing the pin.
|
||||
- A Glassdoor run logs `429` / "Blocked by Glassdoor for too many requests".
|
||||
This is a real rate limit from running too many Glassdoor requests back-to-back (e.g. repeated manual test runs). Space out runs; the automatic pipeline's normal cadence does not trigger this in practice.
|
||||
@@ -25,6 +25,7 @@ Extractor integrations are now registered through manifests and loaded automatic
|
||||
| [Eluta](/docs/next/extractors/eluta) | Canadian listings aggregated from employer career sites (RSS) | Canada-only source (skipped when search geography is not Canada); RSS `location` strings must be set | `ELUTA_RSS_LOCATIONS`, `ELUTA_MAX_JOBS_PER_TERM` | Fetches one or more `eluta.ca` RSS feeds, filters by terms, de-duplicates by guid/URL |
|
||||
| [QAJobsBoard](/docs/next/extractors/qajobsboard) | QA / SDET / automation-heavy board (global JSON feed) | No auth; geography skew is manual/filter downstream | `qajobsboardMaxJobsPerTerm` | Fetches JobBoardly JSON, filters by pipeline terms |
|
||||
| [Arc.dev](/docs/next/extractors/arcdev) | Remote roles from Arc.dev listing pages (tool-tagged paths) | Parses SSR `__NEXT_DATA__`; relies on stable Next payload | `ARC_REMOTE_JOBS_PATHS` (seeds defaults), `arcRemoteJobsPaths`, `arcMaxJobsPerPath` | Merges Arc-managed + external rows; dedupes by URL |
|
||||
| [Google Jobs](/docs/next/extractors/google-jobs) | Broader open-web coverage via the Google for Jobs widget | Unofficial scrape (no API); CAPTCHA-prone from datacenter IPs; selectors can rot; slower (click-through per job) | `googleJobsMaxJobsPerTerm`, `GOOGLE_JOBS_PROXY`, `GOOGLE_JOBS_HEADLESS` | Browser scrape of `udm=8` Jobs vertical; scrolls list, double-clicks cards, expands description |
|
||||
| [Manual Import](/docs/next/extractors/manual) | One-off jobs not covered by scrapers | Inference quality depends on model/provider and input quality; some URLs cannot be fetched reliably | App/API endpoints (`/api/manual-jobs/infer`, `/api/manual-jobs/import`) | Accepts text/HTML/URL, runs inference, then saves and scores job after review |
|
||||
|
||||
## Which extractor should I use?
|
||||
@@ -40,6 +41,7 @@ Extractor integrations are now registered through manifests and loaded automatic
|
||||
- Use **BC T-Net** for British Columbia tech RSS listings (runs only when search geography is Canada).
|
||||
- Use **Eluta** for Canadian employer-direct listings via RSS (set metro/province `location` strings).
|
||||
- Use **QAJobsBoard** or **Arc.dev** when you want QA- or remote-stack-focused feeds without extra credentials.
|
||||
- Use **Google Jobs** when you want broader open-web coverage beyond dedicated job boards (unofficial scrape; best-effort).
|
||||
- Use **Manual Import** when you already have a specific posting and need direct import.
|
||||
|
||||
Many runs combine sources: broad discovery first, then manual import for high-priority jobs that scraping misses.
|
||||
@@ -82,5 +84,7 @@ JobOps ships **BC T-Net** and **iCIMS tenant HTML** extractors for two cases tha
|
||||
- [Arc.dev](/docs/next/extractors/arcdev)
|
||||
- [Canadian / NA QA contracting firms](/docs/next/extractors/qa-contract-staffing-canada)
|
||||
- [Canadian companies — QA-strong ATS](/docs/next/extractors/canadian-companies-qa-ats)
|
||||
- [Google Jobs](/docs/next/extractors/google-jobs)
|
||||
- [Himalayas](/docs/next/extractors/himalayas)
|
||||
- [Manual Import](/docs/next/extractors/manual)
|
||||
- [Add an Extractor](/docs/next/workflows/add-an-extractor)
|
||||
|
||||
Reference in New Issue
Block a user