feat: add Google Jobs extractor and fix scraper reliability gaps
Ship a Camoufox-backed Google Jobs source, restore Glassdoor results discarded by a python-jobspy GraphQL quirk, and fix Himalayas/Gradcracker zero-job failures. Also harden prior-skip dismiss matching and multi-profile basic-auth switching for CA/US runs.
This commit is contained in:
+29
-1
@@ -260,7 +260,35 @@ ADZUNA_APP_KEY=
|
|||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Caps via Settings: workingnomadsMaxJobsPerTerm, testdevjobsMaxJobsPerTerm,
|
# Caps via Settings: workingnomadsMaxJobsPerTerm, testdevjobsMaxJobsPerTerm,
|
||||||
# testdevjobsMaxPages, builtinMaxJobsPerTerm, builtinMaxPagesPerTerm,
|
# testdevjobsMaxPages, builtinMaxJobsPerTerm, builtinMaxPagesPerTerm,
|
||||||
# wellfoundMaxJobsPerTerm (Wellfound uses Camoufox when installed).
|
# wellfoundMaxJobsPerTerm, googleJobsMaxJobsPerTerm (Wellfound and Google
|
||||||
|
# Jobs both use Camoufox when installed).
|
||||||
|
|
||||||
|
# =============================================================================
|
||||||
|
# Google Jobs — optional (auto-enabled, no credentials, unofficial scrape)
|
||||||
|
# =============================================================================
|
||||||
|
# GOOGLE_JOBS_PROXY=http://user:pass@proxy-host:port
|
||||||
|
# ^ Most reliable way to avoid Google's CAPTCHA/rate-limit wall (residential or
|
||||||
|
# mobile proxy recommended; datacenter IPs, including most VPS/cloud hosts,
|
||||||
|
# are aggressively blocked). Applies to both manual runs and the pipeline.
|
||||||
|
#
|
||||||
|
# GOOGLE_JOBS_HEADLESS=false
|
||||||
|
# ^ Manual/local use ONLY (do not set this in the server .env for the
|
||||||
|
# unattended pipeline/cron — a headless server has no display for a human
|
||||||
|
# to see the browser). Opens a real Firefox window so you can solve a
|
||||||
|
# CAPTCHA by hand; waits up to GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS (default
|
||||||
|
# 300000 = 5 min) before giving up. Run directly, e.g.:
|
||||||
|
# GOOGLE_JOBS_HEADLESS=false npm --workspace google-jobs-extractor run start
|
||||||
|
# Session cookies are saved to extractors/google-jobs/storage/state.json
|
||||||
|
# and reused by later runs (including headless/automated ones), which can
|
||||||
|
# reduce how often future runs hit a CAPTCHA.
|
||||||
|
#
|
||||||
|
# Caps via Settings: googleJobsMaxJobsPerTerm (default 30, max 150).
|
||||||
|
#
|
||||||
|
# Note: there is no country/location override for this extractor — Google
|
||||||
|
# resolves location from the browser's real (or GOOGLE_JOBS_PROXY's) IP.
|
||||||
|
# Forcing a mismatched country has been observed to make Google's Jobs
|
||||||
|
# vertical return "no matches" even though the tab loads correctly. To
|
||||||
|
# target a different country's jobs, use a proxy with an exit node there.
|
||||||
|
|
||||||
# =============================================================================
|
# =============================================================================
|
||||||
# Teamtailor / Huntflow / Factorial / Career pages — optional
|
# Teamtailor / Huntflow / Factorial / Career pages — optional
|
||||||
|
|||||||
+2
-1
@@ -12,8 +12,9 @@
|
|||||||
"!!**/.venv",
|
"!!**/.venv",
|
||||||
"!!docs-site/.docusaurus",
|
"!!docs-site/.docusaurus",
|
||||||
"!!docs-site/build",
|
"!!docs-site/build",
|
||||||
"!!extractors/jobspy/storage",
|
"!!extractors/*/storage",
|
||||||
"!!orchestrator/storage",
|
"!!orchestrator/storage",
|
||||||
|
"!!storage",
|
||||||
"!!data"
|
"!!data"
|
||||||
]
|
]
|
||||||
},
|
},
|
||||||
|
|||||||
@@ -32,10 +32,11 @@ Eluta surfaces Canadian roles indexed directly from employer career sites, often
|
|||||||
### Empty feeds
|
### Empty feeds
|
||||||
|
|
||||||
- The `location` string may be too broad or spelled differently than Eluta expects. Try a major city plus province (e.g. `Calgary, AB`).
|
- The `location` string may be too broad or spelled differently than Eluta expects. Try a major city plus province (e.g. `Calgary, AB`).
|
||||||
|
- Eluta's RSS is a small recent slice of *all* jobs for that location (often retail/hospitality-heavy). Pipeline search terms like `software engineer` can legitimately match zero items even when the feed itself is healthy. Broaden terms or add more metro locations.
|
||||||
|
|
||||||
### RSS HTTP errors
|
### RSS HTTP errors / "Too Many Requests"
|
||||||
|
|
||||||
- Eluta may block unusual clients; the extractor sends a conventional User-Agent. Retry later or reduce the number of location feeds per run.
|
- Eluta rate-limits aggressive or browser-like clients. The extractor uses a dedicated `JobOps/... Eluta RSS consumer` User-Agent; retry later or reduce the number of location feeds per run.
|
||||||
|
|
||||||
## Related pages
|
## Related pages
|
||||||
|
|
||||||
|
|||||||
@@ -0,0 +1,106 @@
|
|||||||
|
---
|
||||||
|
id: google-jobs
|
||||||
|
title: Google Jobs Extractor
|
||||||
|
description: Browser-backed scraping of the Google for Jobs search widget.
|
||||||
|
sidebar_position: 8
|
||||||
|
---
|
||||||
|
|
||||||
|
## What it is
|
||||||
|
|
||||||
|
Original feature: [Google for Jobs](https://www.google.com/search?q=jobs&udm=8)
|
||||||
|
|
||||||
|
Google Jobs is a browser-backed extractor that opens Google's Jobs vertical (`udm=8`) in Firefox (Camoufox when available) for each pipeline search term, scrolls the results panel to load additional cards, then clicks into each card to resolve a real "Apply on <site>" link and description before mapping rows into the orchestrator `CreateJobInput` shape.
|
||||||
|
|
||||||
|
Implementation split:
|
||||||
|
|
||||||
|
1. `extractors/google-jobs/src/main.ts` builds the search URL, waits for the jobs list, scrolls to lazy-load more cards, extracts card metadata (title/employer/location/"via" source), clicks each card to read the detail pane, and writes dataset JSON.
|
||||||
|
2. `extractors/google-jobs/src/run.ts` spawns the browser subprocess, applies a hard timeout, reads the dataset, and maps rows to `CreateJobInput`.
|
||||||
|
3. `extractors/google-jobs/manifest.ts` wires pipeline `searchTerms` into the extractor.
|
||||||
|
|
||||||
|
### A note on the search URL and location
|
||||||
|
|
||||||
|
Google's Jobs vertical is reached with `udm=8` (its current internal vertical id). An older, commonly-documented deep link — `ibp=htl;jobs` — still works but gets silently redirected to `udm=8` anyway, and combining it with extra parameters (a `num=` result-count hint, or a `gl=` country override that doesn't match the browser's real/proxied IP) has been observed to make Google return "no matches" or "can't determine location" even though the tab itself loads correctly.
|
||||||
|
|
||||||
|
Because of that, this extractor does **not** send a `gl` (country) parameter — location is resolved by Google from the browser's real IP (or `GOOGLE_JOBS_PROXY`'s exit IP), which Camoufox's `geoip` option keeps internally consistent. To target a different country's jobs, use a residential proxy with an exit node in that country rather than trying to override the country via a parameter.
|
||||||
|
|
||||||
|
### Dealing with Google's CAPTCHA / bot detection
|
||||||
|
|
||||||
|
Google actively fingerprints and challenges automated browser traffic, especially from datacenter/VPS IPs (which most servers running JobOps' cron pipeline will have). The extractor has layered mitigations, from always-on to opt-in:
|
||||||
|
|
||||||
|
1. **Always on:** Camoufox launches with `humanize: true` and `geoip: true` (human-like cursor movement, IP-consistent locale/geolocation) to reduce — not eliminate — fingerprint-based detection.
|
||||||
|
2. **Always on:** browser cookies/session are persisted to `extractors/google-jobs/storage/state.json` and reused on the next run. A session that has cleared a CAPTCHA once (including via option 4 below) tends to see fewer challenges for a while afterward.
|
||||||
|
3. **Opt-in, most reliable:** set `GOOGLE_JOBS_PROXY` (e.g. a residential/mobile proxy URL) to avoid datacenter-IP blocks in the first place. This is the standard real-world fix; unproxied runs from cloud servers should expect to be blocked often.
|
||||||
|
4. **Opt-in, manual/local only:** set `GOOGLE_JOBS_HEADLESS=false` and run the extractor directly (not via the pipeline) to open a real, visible Firefox window. If Google shows a CAPTCHA, solve it yourself in that window — the script polls and automatically resumes once it clears (up to `GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS`, default 5 minutes). The resulting session cookies are saved and reused by later automated/headless runs. **Do not** set `GOOGLE_JOBS_HEADLESS=false` in the server's always-on `.env` — a headless server has no display for a human to interact with, and the run will just wait until the timeout.
|
||||||
|
|
||||||
|
## Why it exists
|
||||||
|
|
||||||
|
Google for Jobs aggregates postings across the open web — including boards and employer career pages that JobOps does not otherwise scrape. Adding it as a proper source closes some of the gap between "jobs found by manually Googling a role" and "jobs discovered by the pipeline."
|
||||||
|
|
||||||
|
It is **not** a general web crawler and does not replace dedicated extractors: it only surfaces whatever Google's Jobs widget chooses to show for a query, subject to Google's own ranking, freshness, and regional availability.
|
||||||
|
|
||||||
|
## How to use it
|
||||||
|
|
||||||
|
1. Open **Run jobs** and choose **Automatic**.
|
||||||
|
2. **Google Jobs** is enabled by default in **Sources** (toggle it off if you do not want it for this run).
|
||||||
|
3. Set your existing automatic run knobs:
|
||||||
|
- `searchTerms` become the `"<term> jobs"` query sent to Google per term.
|
||||||
|
- `googleJobsMaxJobsPerTerm` (Settings) caps jobs scraped per run (default `30`, max `150`).
|
||||||
|
- Location is derived from the browser's real/proxied IP, not from a country setting — see [the note above](#a-note-on-the-search-url-and-location).
|
||||||
|
4. Start the run and watch progress in the pipeline progress card.
|
||||||
|
|
||||||
|
Local run example (headless, no proxy):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
GOOGLE_JOBS_SEARCH_TERMS='["automation engineer"]' \
|
||||||
|
GOOGLE_JOBS_MAX_JOBS='10' \
|
||||||
|
npm --workspace google-jobs-extractor run start
|
||||||
|
```
|
||||||
|
|
||||||
|
Local run example (headed, solve the CAPTCHA yourself if one appears):
|
||||||
|
|
||||||
|
```bash
|
||||||
|
GOOGLE_JOBS_HEADLESS=false \
|
||||||
|
GOOGLE_JOBS_SEARCH_TERMS='["automation engineer"]' \
|
||||||
|
npm --workspace google-jobs-extractor run start
|
||||||
|
```
|
||||||
|
|
||||||
|
Defaults and constraints:
|
||||||
|
|
||||||
|
- No credentials required; this is a direct HTML/DOM scrape of Google's public search results, not an official API (Google does not offer a free Google Jobs API).
|
||||||
|
- Google's markup and CSS class names change frequently and are obfuscated; the scraper favors known current card classes (e.g. `.EimVGf`) with multiple fallback selectors and a leaf-node text heuristic, but selectors can still rot and return zero results until updated.
|
||||||
|
- Every card carries Google's own `data-share-url` permalink for that specific listing, which the extractor uses as `jobUrl` — so a job stays unique and reviewable even if the "click into the card for a detail pane" step below fails for it.
|
||||||
|
- Because resolving a real application link requires clicking each card in a real browser, this extractor is slower per job than API-backed sources — keep `googleJobsMaxJobsPerTerm` modest for frequent runs.
|
||||||
|
- If no external "Apply on <site>" link is found for a card, the extractor falls back to Google's own permalink for that listing so it's still reviewable, but not directly one-click-applyable.
|
||||||
|
- Automated, frequent, or high-volume scraping of Google search results may be against Google's Terms of Service; treat this extractor as best-effort and keep run frequency/volume conservative.
|
||||||
|
- See [Dealing with Google's CAPTCHA / bot detection](#dealing-with-googles-captcha--bot-detection) above for proxy and interactive-solve options.
|
||||||
|
|
||||||
|
## Common problems
|
||||||
|
|
||||||
|
### Google Jobs returns 0 jobs for a term
|
||||||
|
|
||||||
|
The extractor logs one of two distinct messages to help tell these apart:
|
||||||
|
|
||||||
|
- `Google's Jobs tab reported no matches for "<term>"` — Google correctly landed on the Jobs tab but genuinely has no results for that query/locale. Try a broader, more common phrasing (e.g. `"automation engineer"` instead of a very narrow title).
|
||||||
|
- `Jobs list did not appear for "<term>"` — an unrecognized page state (consent wall, layout change, or rotted selectors). The extractor saves a screenshot and full HTML dump to `extractors/google-jobs/storage/debug-last.png` / `debug-last.html` for inspection; compare it against a fresh manual search to see what changed.
|
||||||
|
|
||||||
|
### Results look sparse or missing company/location
|
||||||
|
|
||||||
|
- The list-view heuristics rely on leaf DOM nodes near the job title; if Google restructures the card markup, company/location extraction can degrade even when titles still work. This is expected best-effort behavior for an unofficial scrape.
|
||||||
|
|
||||||
|
### Run is slow
|
||||||
|
|
||||||
|
- Each job requires a real click + detail-pane render, unlike single-request API extractors. Lower `googleJobsMaxJobsPerTerm` or reduce the number of search terms for faster runs.
|
||||||
|
|
||||||
|
### Blocked, CAPTCHA'd, or consistently empty results
|
||||||
|
|
||||||
|
- Google may rate-limit or challenge automated browser traffic, especially from datacenter/VPS IPs. In order of effort:
|
||||||
|
1. Set `GOOGLE_JOBS_PROXY` to a residential/mobile proxy — the most reliable fix for unattended/server runs.
|
||||||
|
2. Run locally with `GOOGLE_JOBS_HEADLESS=false` and solve the CAPTCHA yourself once; the saved session (`storage/state.json`) may reduce blocks on subsequent headless runs for a while.
|
||||||
|
3. Reduce run frequency, lower the per-term cap, or disable the source for that run and rely on other extractors / Manual Import instead.
|
||||||
|
|
||||||
|
## Related pages
|
||||||
|
|
||||||
|
- [Extractors Overview](/docs/next/extractors/overview)
|
||||||
|
- [Manual Import](/docs/next/extractors/manual)
|
||||||
|
- [Pipeline Run](/docs/next/features/pipeline-run)
|
||||||
|
- [Settings](/docs/next/features/settings)
|
||||||
@@ -46,3 +46,11 @@ Set `JOBOPS_EMIT_PROGRESS=1` for structured progress lines consumable by orchest
|
|||||||
|
|
||||||
- Uses Playwright + Crawlee via Camoufox.
|
- Uses Playwright + Crawlee via Camoufox.
|
||||||
- Low concurrency and longer timeouts for stability.
|
- Low concurrency and longer timeouts for stability.
|
||||||
|
- Employer logo (`figure img` alt text) and employer link are optional. Cards that omit them used to throw a Playwright `getAttribute` timeout and fail the whole region page (0 jobs saved even after cards were counted). Missing employer fields are now skipped per card.
|
||||||
|
|
||||||
|
## Common problems
|
||||||
|
|
||||||
|
### Region pages time out / run returns 0 jobs
|
||||||
|
|
||||||
|
- Gradcracker Livewire markup sometimes omits the employer logo on a card. Older builds aborted the whole page on that timeout. Current builds continue and leave employer blank.
|
||||||
|
- Anti-bot / Camoufox issues: ensure Camoufox is installed (`npx camoufox-js fetch`) and prefer headed runs when debugging.
|
||||||
|
|||||||
@@ -0,0 +1,45 @@
|
|||||||
|
---
|
||||||
|
id: himalayas
|
||||||
|
title: Himalayas Extractor
|
||||||
|
description: Public remote-jobs API pagination and client-side term filtering.
|
||||||
|
sidebar_position: 16
|
||||||
|
---
|
||||||
|
|
||||||
|
## What it is
|
||||||
|
|
||||||
|
Original site: [himalayas.app](https://himalayas.app)
|
||||||
|
|
||||||
|
The extractor lives in `extractors/himalayas/manifest.ts`. It paginates the public JSON API (`https://himalayas.app/jobs/api?limit=&offset=`), filters rows client-side by pipeline search terms (title + categories), and maps matches into `CreateJobInput`.
|
||||||
|
|
||||||
|
## Why it exists
|
||||||
|
|
||||||
|
Himalayas is a large remote-jobs index with a stable, unauthenticated API — useful for remote-first discovery without browser automation or credentials.
|
||||||
|
|
||||||
|
## How to use it
|
||||||
|
|
||||||
|
1. Enable **Himalayas** in pipeline sources.
|
||||||
|
2. Optionally set **Himalayas max jobs per term** (`himalayasMaxJobsPerTerm`, default `100`).
|
||||||
|
3. Pipeline `searchTerms` filter titles/categories client-side (substring match, case-insensitive).
|
||||||
|
4. Run the pipeline.
|
||||||
|
|
||||||
|
### Defaults and constraints
|
||||||
|
|
||||||
|
- The upstream API **silently caps `limit` at 20** per request. The extractor uses `PAGE_SIZE = 20` and paginates with `offset` (up to 10 pages / 200 rows scanned per run before the per-term cap).
|
||||||
|
- There is no server-side search — if a term is rare, early pages may not contain matches even though later pages do. Raise the page budget only by changing the extractor constants if you need deeper scans.
|
||||||
|
- No auth required.
|
||||||
|
|
||||||
|
## Common problems
|
||||||
|
|
||||||
|
### Zero jobs for a term I know exists
|
||||||
|
|
||||||
|
- The term may not appear in the first ~200 API rows (newest-first feed). Try a shorter token (`software`, `engineer`, `SDET`) or accept that Himalayas skews toward whatever is currently at the top of the feed.
|
||||||
|
- Confirm you are not hitting an old build that requested `limit=50` and stopped after one page (fixed: the API returns at most 20, which used to look like end-of-feed).
|
||||||
|
|
||||||
|
### Rate limits / HTTP errors
|
||||||
|
|
||||||
|
- Retry later; the public API is generally stable but can throttle burst traffic.
|
||||||
|
|
||||||
|
## Related pages
|
||||||
|
|
||||||
|
- [Extractors Overview](/docs/next/extractors/overview)
|
||||||
|
- [Add an Extractor](/docs/next/workflows/add-an-extractor)
|
||||||
@@ -62,3 +62,7 @@ The service in `orchestrator/src/server/services/jobspy.ts`:
|
|||||||
`JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0` disables description fetching to speed up runs.
|
`JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0` disables description fetching to speed up runs.
|
||||||
- Different cities need different workplace-type filters.
|
- Different cities need different workplace-type filters.
|
||||||
This is not supported in the current automatic-run flow. JobSpy receives one global workplace-type selection per run/query invocation.
|
This is not supported in the current automatic-run flow. JobSpy receives one global workplace-type selection per run/query invocation.
|
||||||
|
- Glassdoor always returns 0 jobs (fixed).
|
||||||
|
Upstream `python-jobspy` (through at least `1.1.82`) discards an entire page of Glassdoor results whenever Glassdoor's GraphQL response includes *any* `errors` entry — even for an unrelated field. In practice, Glassdoor's backend consistently returns a partial error for `jobsPageSeoData` (`SUBREQUEST_HTTP_ERROR` / "dns error" on Glassdoor's own infra) while the actual job listings still come back fine, so every Glassdoor run silently returned 0 jobs. `extractors/jobspy/scrape_jobs.py` monkeypatches `Glassdoor._fetch_jobs_page` at import time to only treat the response as failed when `data.jobListings` is actually missing, and logs a warning (not an error) when it ignores an unrelated partial error. `requirements.txt` pins `python-jobspy==1.1.82` so an upstream version bump doesn't silently drop this patch — re-verify the patched method still matches before removing the pin.
|
||||||
|
- A Glassdoor run logs `429` / "Blocked by Glassdoor for too many requests".
|
||||||
|
This is a real rate limit from running too many Glassdoor requests back-to-back (e.g. repeated manual test runs). Space out runs; the automatic pipeline's normal cadence does not trigger this in practice.
|
||||||
@@ -25,6 +25,7 @@ Extractor integrations are now registered through manifests and loaded automatic
|
|||||||
| [Eluta](/docs/next/extractors/eluta) | Canadian listings aggregated from employer career sites (RSS) | Canada-only source (skipped when search geography is not Canada); RSS `location` strings must be set | `ELUTA_RSS_LOCATIONS`, `ELUTA_MAX_JOBS_PER_TERM` | Fetches one or more `eluta.ca` RSS feeds, filters by terms, de-duplicates by guid/URL |
|
| [Eluta](/docs/next/extractors/eluta) | Canadian listings aggregated from employer career sites (RSS) | Canada-only source (skipped when search geography is not Canada); RSS `location` strings must be set | `ELUTA_RSS_LOCATIONS`, `ELUTA_MAX_JOBS_PER_TERM` | Fetches one or more `eluta.ca` RSS feeds, filters by terms, de-duplicates by guid/URL |
|
||||||
| [QAJobsBoard](/docs/next/extractors/qajobsboard) | QA / SDET / automation-heavy board (global JSON feed) | No auth; geography skew is manual/filter downstream | `qajobsboardMaxJobsPerTerm` | Fetches JobBoardly JSON, filters by pipeline terms |
|
| [QAJobsBoard](/docs/next/extractors/qajobsboard) | QA / SDET / automation-heavy board (global JSON feed) | No auth; geography skew is manual/filter downstream | `qajobsboardMaxJobsPerTerm` | Fetches JobBoardly JSON, filters by pipeline terms |
|
||||||
| [Arc.dev](/docs/next/extractors/arcdev) | Remote roles from Arc.dev listing pages (tool-tagged paths) | Parses SSR `__NEXT_DATA__`; relies on stable Next payload | `ARC_REMOTE_JOBS_PATHS` (seeds defaults), `arcRemoteJobsPaths`, `arcMaxJobsPerPath` | Merges Arc-managed + external rows; dedupes by URL |
|
| [Arc.dev](/docs/next/extractors/arcdev) | Remote roles from Arc.dev listing pages (tool-tagged paths) | Parses SSR `__NEXT_DATA__`; relies on stable Next payload | `ARC_REMOTE_JOBS_PATHS` (seeds defaults), `arcRemoteJobsPaths`, `arcMaxJobsPerPath` | Merges Arc-managed + external rows; dedupes by URL |
|
||||||
|
| [Google Jobs](/docs/next/extractors/google-jobs) | Broader open-web coverage via the Google for Jobs widget | Unofficial scrape (no API); CAPTCHA-prone from datacenter IPs; selectors can rot; slower (click-through per job) | `googleJobsMaxJobsPerTerm`, `GOOGLE_JOBS_PROXY`, `GOOGLE_JOBS_HEADLESS` | Browser scrape of `udm=8` Jobs vertical; scrolls list, double-clicks cards, expands description |
|
||||||
| [Manual Import](/docs/next/extractors/manual) | One-off jobs not covered by scrapers | Inference quality depends on model/provider and input quality; some URLs cannot be fetched reliably | App/API endpoints (`/api/manual-jobs/infer`, `/api/manual-jobs/import`) | Accepts text/HTML/URL, runs inference, then saves and scores job after review |
|
| [Manual Import](/docs/next/extractors/manual) | One-off jobs not covered by scrapers | Inference quality depends on model/provider and input quality; some URLs cannot be fetched reliably | App/API endpoints (`/api/manual-jobs/infer`, `/api/manual-jobs/import`) | Accepts text/HTML/URL, runs inference, then saves and scores job after review |
|
||||||
|
|
||||||
## Which extractor should I use?
|
## Which extractor should I use?
|
||||||
@@ -40,6 +41,7 @@ Extractor integrations are now registered through manifests and loaded automatic
|
|||||||
- Use **BC T-Net** for British Columbia tech RSS listings (runs only when search geography is Canada).
|
- Use **BC T-Net** for British Columbia tech RSS listings (runs only when search geography is Canada).
|
||||||
- Use **Eluta** for Canadian employer-direct listings via RSS (set metro/province `location` strings).
|
- Use **Eluta** for Canadian employer-direct listings via RSS (set metro/province `location` strings).
|
||||||
- Use **QAJobsBoard** or **Arc.dev** when you want QA- or remote-stack-focused feeds without extra credentials.
|
- Use **QAJobsBoard** or **Arc.dev** when you want QA- or remote-stack-focused feeds without extra credentials.
|
||||||
|
- Use **Google Jobs** when you want broader open-web coverage beyond dedicated job boards (unofficial scrape; best-effort).
|
||||||
- Use **Manual Import** when you already have a specific posting and need direct import.
|
- Use **Manual Import** when you already have a specific posting and need direct import.
|
||||||
|
|
||||||
Many runs combine sources: broad discovery first, then manual import for high-priority jobs that scraping misses.
|
Many runs combine sources: broad discovery first, then manual import for high-priority jobs that scraping misses.
|
||||||
@@ -82,5 +84,7 @@ JobOps ships **BC T-Net** and **iCIMS tenant HTML** extractors for two cases tha
|
|||||||
- [Arc.dev](/docs/next/extractors/arcdev)
|
- [Arc.dev](/docs/next/extractors/arcdev)
|
||||||
- [Canadian / NA QA contracting firms](/docs/next/extractors/qa-contract-staffing-canada)
|
- [Canadian / NA QA contracting firms](/docs/next/extractors/qa-contract-staffing-canada)
|
||||||
- [Canadian companies — QA-strong ATS](/docs/next/extractors/canadian-companies-qa-ats)
|
- [Canadian companies — QA-strong ATS](/docs/next/extractors/canadian-companies-qa-ats)
|
||||||
|
- [Google Jobs](/docs/next/extractors/google-jobs)
|
||||||
|
- [Himalayas](/docs/next/extractors/himalayas)
|
||||||
- [Manual Import](/docs/next/extractors/manual)
|
- [Manual Import](/docs/next/extractors/manual)
|
||||||
- [Add an Extractor](/docs/next/workflows/add-an-extractor)
|
- [Add an Extractor](/docs/next/workflows/add-an-extractor)
|
||||||
|
|||||||
@@ -45,9 +45,19 @@ During a pipeline run, if a new posting matches an existing row by URL, source i
|
|||||||
|
|
||||||
Open jobs that match a prior skip or apply are **hidden** from Discovered, Ready, and All tabs so the queue stays fresh. Skipped and applied rows themselves remain visible in their statuses.
|
Open jobs that match a prior skip or apply are **hidden** from Discovered, Ready, and All tabs so the queue stays fresh. Skipped and applied rows themselves remain visible in their statuses.
|
||||||
|
|
||||||
|
Use **Filters → Employer keywords → Hide roles you already skipped** to toggle this behavior (on by default). When enabled, JobOps compares **company + job title** only:
|
||||||
|
|
||||||
|
- Same company after normalization (legal suffixes stripped; short names like `CGI` match longer forms like `CGI IT UK Limited`).
|
||||||
|
- Same title after normalization (including simple plural variants such as `Engineer` vs `Engineers`).
|
||||||
|
- Prior skip or apply must be within the **last 90 days** (`updatedAt` on the skipped/applied row).
|
||||||
|
|
||||||
|
Uncheck the filter to temporarily review rediscovered rows that match older skips.
|
||||||
|
|
||||||
## Defaults and constraints
|
## Defaults and constraints
|
||||||
|
|
||||||
- Description matching requires at least **80 characters** of normalized text; short or empty descriptions fall back to employer+title only.
|
- Prior-skip hiding uses employer + title only (not description text).
|
||||||
|
- The 90-day window applies to both the Jobs list filter and pipeline import suppression for skipped/applied rows.
|
||||||
|
- Description matching during import still requires at least **80 characters** of normalized text; short or empty descriptions fall back to employer+title only.
|
||||||
- Matching is **per profile** (`ownerProfileId`); different login profiles do not share dedup state.
|
- Matching is **per profile** (`ownerProfileId`); different login profiles do not share dedup state.
|
||||||
- Dedup does **not** delete existing rows retroactively when you change skip list or country filters — run discovery again or skip manually for old data.
|
- Dedup does **not** delete existing rows retroactively when you change skip list or country filters — run discovery again or skip manually for old data.
|
||||||
- Very different titles at the same company (for example `SDET` vs `Product Designer`) are **not** collapsed.
|
- Very different titles at the same company (for example `SDET` vs `Product Designer`) are **not** collapsed.
|
||||||
|
|||||||
@@ -0,0 +1,49 @@
|
|||||||
|
import type {
|
||||||
|
ExtractorManifest,
|
||||||
|
ExtractorRunResult,
|
||||||
|
} from "@shared/types/extractors";
|
||||||
|
import { runGoogleJobs } from "./src/run.js";
|
||||||
|
|
||||||
|
export const manifest: ExtractorManifest = {
|
||||||
|
id: "google-jobs",
|
||||||
|
displayName: "Google Jobs",
|
||||||
|
providesSources: ["google-jobs"],
|
||||||
|
async run(context): Promise<ExtractorRunResult> {
|
||||||
|
if (context.shouldCancel?.()) return { success: true, jobs: [] };
|
||||||
|
|
||||||
|
const parsedMax = context.settings.googleJobsMaxJobsPerTerm
|
||||||
|
? Number.parseInt(context.settings.googleJobsMaxJobsPerTerm, 10)
|
||||||
|
: Number.NaN;
|
||||||
|
const maxJobs = Number.isFinite(parsedMax) ? Math.max(1, parsedMax) : 30;
|
||||||
|
|
||||||
|
context.onProgress?.({
|
||||||
|
phase: "list",
|
||||||
|
termsProcessed: 0,
|
||||||
|
termsTotal: context.searchTerms.length || 1,
|
||||||
|
currentUrl: "https://www.google.com/search?udm=8",
|
||||||
|
detail: "Google Jobs: launching browser scrape",
|
||||||
|
});
|
||||||
|
|
||||||
|
const result = await runGoogleJobs({
|
||||||
|
searchTerms: context.searchTerms,
|
||||||
|
maxJobs,
|
||||||
|
});
|
||||||
|
|
||||||
|
if (!result.success) {
|
||||||
|
return { success: false, jobs: [], error: result.error };
|
||||||
|
}
|
||||||
|
|
||||||
|
context.onProgress?.({
|
||||||
|
phase: "list",
|
||||||
|
termsProcessed: context.searchTerms.length || 1,
|
||||||
|
termsTotal: context.searchTerms.length || 1,
|
||||||
|
currentUrl: "https://www.google.com/search?udm=8",
|
||||||
|
jobPagesProcessed: result.jobs.length,
|
||||||
|
detail: `Google Jobs: ${result.jobs.length} jobs`,
|
||||||
|
});
|
||||||
|
|
||||||
|
return { success: true, jobs: result.jobs };
|
||||||
|
},
|
||||||
|
};
|
||||||
|
|
||||||
|
export default manifest;
|
||||||
@@ -0,0 +1,22 @@
|
|||||||
|
{
|
||||||
|
"name": "google-jobs-extractor",
|
||||||
|
"version": "0.0.1",
|
||||||
|
"type": "module",
|
||||||
|
"description": "Google Jobs (Google for Jobs) search widget extractor (browser-backed)",
|
||||||
|
"main": "manifest.ts",
|
||||||
|
"dependencies": {
|
||||||
|
"camoufox-js": "^0.8.0",
|
||||||
|
"job-ops-shared": "^1.0.0",
|
||||||
|
"playwright": "^1.57.0",
|
||||||
|
"tsx": "^4.4.0"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@types/node": "^24.0.0",
|
||||||
|
"typescript": "~5.9.0"
|
||||||
|
},
|
||||||
|
"scripts": {
|
||||||
|
"start": "tsx src/main.ts",
|
||||||
|
"check:types": "tsc --noEmit",
|
||||||
|
"get-binaries": "camoufox-js fetch"
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,616 @@
|
|||||||
|
import { access, mkdir, writeFile } from "node:fs/promises";
|
||||||
|
import { dirname, join } from "node:path";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
import { launchOptions } from "camoufox-js";
|
||||||
|
import {
|
||||||
|
type Browser,
|
||||||
|
type BrowserContext,
|
||||||
|
firefox,
|
||||||
|
type Page,
|
||||||
|
} from "playwright";
|
||||||
|
|
||||||
|
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const OUTPUT_PATH = join(__dirname, "../storage/jobs.json");
|
||||||
|
const STATE_PATH = join(__dirname, "../storage/state.json");
|
||||||
|
const DEBUG_HTML_PATH = join(__dirname, "../storage/debug-last.html");
|
||||||
|
const DEBUG_SCREENSHOT_PATH = join(__dirname, "../storage/debug-last.png");
|
||||||
|
|
||||||
|
interface ScrapedJob {
|
||||||
|
title: string;
|
||||||
|
employer: string;
|
||||||
|
jobUrl: string;
|
||||||
|
applicationLink?: string;
|
||||||
|
location?: string;
|
||||||
|
jobDescription?: string;
|
||||||
|
postedVia?: string;
|
||||||
|
isRemote?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface ScrapedCard {
|
||||||
|
docId?: string;
|
||||||
|
title: string;
|
||||||
|
employer: string;
|
||||||
|
location?: string;
|
||||||
|
via?: string;
|
||||||
|
/** Google's own permalink into this specific listing (from `data-share-url`). Always present when the card matched, so it works as a stable jobUrl even if the detail click fails. */
|
||||||
|
shareUrl?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
interface ScrapedDetail {
|
||||||
|
applyLinks: Array<{ href: string; text: string }>;
|
||||||
|
description?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Confirmed against Google's live markup (2026). Google renders each job as
|
||||||
|
* a `.EimVGf` card with a `data-share-url` permalink baked in — no `<ul>`/
|
||||||
|
* `<li>`/`aria-label="Jobs list"` wrapper as older scraping guides assumed.
|
||||||
|
* Kept as a short list (not a single hardcoded class) since Google rotates
|
||||||
|
* these obfuscated names periodically.
|
||||||
|
*/
|
||||||
|
const CARD_SELECTORS = [".EimVGf", 'li[role="listitem"]', ".iFjolb"] as const;
|
||||||
|
|
||||||
|
function parseTerms(raw: string | undefined): string[] {
|
||||||
|
if (!raw) return ["software engineer"];
|
||||||
|
try {
|
||||||
|
const parsed = JSON.parse(raw) as unknown;
|
||||||
|
if (Array.isArray(parsed)) {
|
||||||
|
return parsed.map((entry) => String(entry).trim()).filter(Boolean);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// fall through to delimited parsing
|
||||||
|
}
|
||||||
|
return raw
|
||||||
|
.split(/[\n|,]+/)
|
||||||
|
.map((entry) => entry.trim())
|
||||||
|
.filter(Boolean);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* `udm=8` is Google's current internal id for the Jobs vertical (the legacy
|
||||||
|
* `ibp=htl;jobs` deep link now gets silently redirected to it anyway, but
|
||||||
|
* only when requested directly — combining the two, or adding `num=`/`gl=`
|
||||||
|
* overrides that don't match the browser's real/proxied IP, has been
|
||||||
|
* observed to make the vertical come back "no matches" or "can't determine
|
||||||
|
* location" even though the widget itself loads correctly). `hl=en` keeps
|
||||||
|
* copy in English so downstream text parsing (e.g. "via LinkedIn") is
|
||||||
|
* consistent regardless of where the request is geolocated. Location is
|
||||||
|
* intentionally NOT set via a `gl` param — it's derived from the real (or
|
||||||
|
* `GOOGLE_JOBS_PROXY`) IP through Camoufox's `geoip` option instead.
|
||||||
|
*/
|
||||||
|
function buildSearchUrl(term: string): string {
|
||||||
|
const params = new URLSearchParams({
|
||||||
|
q: `${term} jobs`,
|
||||||
|
udm: "8",
|
||||||
|
hl: "en",
|
||||||
|
});
|
||||||
|
return `https://www.google.com/search?${params.toString()}`;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Launches Camoufox-hardened Firefox. `humanize`/`geoip` reduce (but do not
|
||||||
|
* eliminate) Google's automated-traffic detection. `GOOGLE_JOBS_PROXY` (e.g.
|
||||||
|
* a residential proxy URL) is the most reliable way to avoid datacenter-IP
|
||||||
|
* blocks entirely, if configured.
|
||||||
|
*/
|
||||||
|
async function launchBrowser(headless: boolean): Promise<Browser> {
|
||||||
|
const proxy = process.env.GOOGLE_JOBS_PROXY?.trim();
|
||||||
|
return firefox.launch(
|
||||||
|
await launchOptions({
|
||||||
|
headless,
|
||||||
|
humanize: true,
|
||||||
|
geoip: true,
|
||||||
|
...(proxy ? { proxy } : {}),
|
||||||
|
}),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Google shows a cookie/consent wall in some locales before rendering results. */
|
||||||
|
async function dismissConsent(page: Page): Promise<void> {
|
||||||
|
const labels = ["Reject all", "I agree", "Accept all"];
|
||||||
|
for (const label of labels) {
|
||||||
|
try {
|
||||||
|
const button = page.getByRole("button", { name: label }).first();
|
||||||
|
if (await button.isVisible({ timeout: 1500 })) {
|
||||||
|
await button.click({ timeout: 3000 });
|
||||||
|
await page.waitForTimeout(500);
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// selector not present in this render; try the next label
|
||||||
|
}
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Thrown when Google serves a CAPTCHA / rate-limit challenge instead of results. */
|
||||||
|
class GoogleBlockedError extends Error {}
|
||||||
|
|
||||||
|
/** Google serves an interstitial CAPTCHA page for automated / datacenter traffic. */
|
||||||
|
async function isBlockedPage(page: Page): Promise<boolean> {
|
||||||
|
if (/\/sorry\//.test(page.url())) return true;
|
||||||
|
try {
|
||||||
|
return await page.evaluate(() => {
|
||||||
|
const bodyText = document.body?.textContent ?? "";
|
||||||
|
return (
|
||||||
|
document.querySelector("#captcha-form") !== null ||
|
||||||
|
document.querySelector(".g-recaptcha") !== null ||
|
||||||
|
/unusual traffic/i.test(bodyText)
|
||||||
|
);
|
||||||
|
});
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* In headed/interactive mode, give a human at the keyboard a chance to solve
|
||||||
|
* the CAPTCHA in the visible browser window instead of failing immediately.
|
||||||
|
* Polls until the challenge clears (Google's form redirects on success) or
|
||||||
|
* the timeout elapses.
|
||||||
|
*/
|
||||||
|
async function waitForHumanToUnblock(
|
||||||
|
page: Page,
|
||||||
|
timeoutMs: number,
|
||||||
|
): Promise<boolean> {
|
||||||
|
const deadline = Date.now() + timeoutMs;
|
||||||
|
console.error(
|
||||||
|
`[google-jobs] Google is showing a CAPTCHA in the open browser window — please solve it there. Waiting up to ${Math.round(timeoutMs / 1000)}s...`,
|
||||||
|
);
|
||||||
|
while (Date.now() < deadline) {
|
||||||
|
await page.waitForTimeout(3_000);
|
||||||
|
if (!(await isBlockedPage(page))) {
|
||||||
|
console.error("[google-jobs] CAPTCHA cleared, resuming.");
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function waitForJobsList(page: Page): Promise<boolean> {
|
||||||
|
for (const selector of CARD_SELECTORS) {
|
||||||
|
try {
|
||||||
|
await page.waitForSelector(selector, { timeout: 8_000 });
|
||||||
|
return true;
|
||||||
|
} catch {
|
||||||
|
// try the next fallback selector
|
||||||
|
}
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Google sometimes lands correctly on the Jobs vertical/tab but reports no
|
||||||
|
* matches for the query (distinct from a CAPTCHA block or a selector-rot
|
||||||
|
* failure) — e.g. "It looks like there aren't any 'Jobs' matches on this topic."
|
||||||
|
*/
|
||||||
|
async function isEmptyJobsVertical(page: Page): Promise<boolean> {
|
||||||
|
try {
|
||||||
|
return await page.evaluate(() => {
|
||||||
|
const bodyText = document.body?.textContent ?? "";
|
||||||
|
return /aren.t any .Jobs. matches/i.test(bodyText);
|
||||||
|
});
|
||||||
|
} catch {
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function countCardsScript(selectors: readonly string[]): number {
|
||||||
|
for (const selector of selectors) {
|
||||||
|
const count = document.querySelectorAll(selector).length;
|
||||||
|
if (count > 0) return count;
|
||||||
|
}
|
||||||
|
return 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Google lazy-loads additional cards (via a custom `<infinity-scrolling>` element) as the page is scrolled. */
|
||||||
|
function scrollPageScript(): number {
|
||||||
|
const before = window.scrollY;
|
||||||
|
window.scrollTo(0, document.body.scrollHeight);
|
||||||
|
return window.scrollY - before;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function growJobsList(
|
||||||
|
page: Page,
|
||||||
|
selectors: readonly string[],
|
||||||
|
targetCount: number,
|
||||||
|
): Promise<void> {
|
||||||
|
for (let attempt = 0; attempt < 8; attempt += 1) {
|
||||||
|
const count = await page.evaluate(countCardsScript, selectors);
|
||||||
|
if (count >= targetCount) return;
|
||||||
|
const scrolled = await page.evaluate(scrollPageScript);
|
||||||
|
if (scrolled <= 0) return;
|
||||||
|
await page.waitForTimeout(900);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function extractCardsScript(selectors: readonly string[]): ScrapedCard[] {
|
||||||
|
// NOTE: this whole function is serialized via Function.prototype.toString()
|
||||||
|
// and re-run inside the browser page by Playwright — it cannot reference
|
||||||
|
// anything from the outer module scope (constants, other functions, etc.).
|
||||||
|
// All helpers must be declared inside this function body.
|
||||||
|
function clean(value: string | null | undefined): string {
|
||||||
|
return (value ?? "").replace(/\s+/g, " ").trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Extracts Google's internal `htidocid` from a job's `data-share-url`, matching `data-encoded-docid` in the detail panel's apply-link list. */
|
||||||
|
function extractDocId(shareUrl: string | null): string | undefined {
|
||||||
|
if (!shareUrl) return undefined;
|
||||||
|
try {
|
||||||
|
const url = new URL(shareUrl, "https://www.google.com");
|
||||||
|
return url.searchParams.get("htidocid") ?? undefined;
|
||||||
|
} catch {
|
||||||
|
return undefined;
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
function findCards(): Element[] {
|
||||||
|
for (const selector of selectors) {
|
||||||
|
const items = Array.from(document.querySelectorAll(selector));
|
||||||
|
if (items.length > 0) return items;
|
||||||
|
}
|
||||||
|
return [];
|
||||||
|
}
|
||||||
|
|
||||||
|
const cards: ScrapedCard[] = [];
|
||||||
|
|
||||||
|
for (const item of findCards()) {
|
||||||
|
// Known current classes first (title/employer/location+via), falling
|
||||||
|
// back to a leaf-node text heuristic if Google has rotated them.
|
||||||
|
const titleEl = item.querySelector(".tNxQIb, h2, [role='heading']");
|
||||||
|
const title = clean(titleEl?.textContent);
|
||||||
|
if (!title) continue;
|
||||||
|
|
||||||
|
const wHYlTdEls = Array.from(item.querySelectorAll(".wHYlTd"));
|
||||||
|
let employer = clean(wHYlTdEls[0]?.textContent);
|
||||||
|
let location: string | undefined;
|
||||||
|
let via: string | undefined;
|
||||||
|
|
||||||
|
const viaBlock = wHYlTdEls.find((el) =>
|
||||||
|
/\bvia\b/i.test(el.textContent ?? ""),
|
||||||
|
);
|
||||||
|
if (viaBlock) {
|
||||||
|
const text = clean(viaBlock.textContent);
|
||||||
|
const match = text.match(/^(.*?)\s*(?:•|·)\s*via\s+(.+)$/i);
|
||||||
|
if (match) {
|
||||||
|
location = match[1] || undefined;
|
||||||
|
via = match[2] || undefined;
|
||||||
|
} else {
|
||||||
|
via = clean(text.replace(/^via\s*/i, ""));
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
if (!employer || !title || employer === title) {
|
||||||
|
// Selector rot fallback: scan leaf text nodes structurally.
|
||||||
|
const leafTexts = Array.from(item.querySelectorAll("div, span"))
|
||||||
|
.filter((el) => el.children.length === 0)
|
||||||
|
.map((el) => clean(el.textContent))
|
||||||
|
.filter((text) => text && text !== title && !/^[•·|,]+$/.test(text));
|
||||||
|
employer = employer || leafTexts[0] || "Unknown Employer";
|
||||||
|
if (!via) {
|
||||||
|
const viaText = leafTexts.find((text) => /^via\b/i.test(text));
|
||||||
|
via = viaText ? clean(viaText.replace(/^via\s*/i, "")) : undefined;
|
||||||
|
location =
|
||||||
|
location ?? leafTexts.slice(1).find((text) => text !== viaText);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const shareUrlRaw = item.getAttribute("data-share-url") ?? undefined;
|
||||||
|
const docId = extractDocId(shareUrlRaw ?? null);
|
||||||
|
|
||||||
|
cards.push({
|
||||||
|
docId,
|
||||||
|
title,
|
||||||
|
employer: employer || "Unknown Employer",
|
||||||
|
location,
|
||||||
|
via,
|
||||||
|
shareUrl: shareUrlRaw,
|
||||||
|
});
|
||||||
|
}
|
||||||
|
|
||||||
|
return cards;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Scopes apply-link extraction to the clicked job's own `data-encoded-docid`
|
||||||
|
* block. Google's detail viewer can have adjacent/related jobs' apply links
|
||||||
|
* present in the DOM simultaneously, so a document-wide "first apply link"
|
||||||
|
* scan can silently pick up the wrong job's link.
|
||||||
|
*/
|
||||||
|
function extractDetailScript(docId: string | undefined): ScrapedDetail {
|
||||||
|
function clean(value: string | null | undefined): string {
|
||||||
|
return (value ?? "").replace(/\s+/g, " ").trim();
|
||||||
|
}
|
||||||
|
|
||||||
|
function collectApplyLinks(
|
||||||
|
root: ParentNode,
|
||||||
|
): Array<{ href: string; text: string }> {
|
||||||
|
const links: Array<{ href: string; text: string }> = [];
|
||||||
|
for (const anchor of Array.from(root.querySelectorAll("a[href]"))) {
|
||||||
|
const href = (anchor as HTMLAnchorElement).href;
|
||||||
|
const text =
|
||||||
|
clean(anchor.textContent) || clean(anchor.getAttribute("aria-label"));
|
||||||
|
if (!href) continue;
|
||||||
|
if (
|
||||||
|
!/apply/i.test(text) &&
|
||||||
|
!/apply/i.test(anchor.getAttribute("aria-label") ?? "")
|
||||||
|
) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
href.includes("google.com/search") ||
|
||||||
|
href.includes("accounts.google.com")
|
||||||
|
) {
|
||||||
|
continue;
|
||||||
|
}
|
||||||
|
links.push({ href, text });
|
||||||
|
}
|
||||||
|
return links;
|
||||||
|
}
|
||||||
|
|
||||||
|
let applyLinks: Array<{ href: string; text: string }> = [];
|
||||||
|
if (docId) {
|
||||||
|
const scoped = document.querySelector(`[data-encoded-docid="${docId}"]`);
|
||||||
|
if (scoped) applyLinks = collectApplyLinks(scoped);
|
||||||
|
}
|
||||||
|
if (applyLinks.length === 0) {
|
||||||
|
applyLinks = collectApplyLinks(document);
|
||||||
|
}
|
||||||
|
|
||||||
|
// `.textContent` also pulls in text from nested <style>/<script> tags
|
||||||
|
// (Google embeds scoped theme <style> blocks inside content divs) and from
|
||||||
|
// the job-list sidebar itself (which concatenates every visible card's
|
||||||
|
// text and easily outsizes a real description) — exclude both.
|
||||||
|
const blocks = Array.from(document.querySelectorAll("div"))
|
||||||
|
.filter(
|
||||||
|
(el) =>
|
||||||
|
el.children.length <= 3 && !el.querySelector("style, script, .EimVGf"),
|
||||||
|
)
|
||||||
|
.map((el) => clean(el.textContent))
|
||||||
|
.filter((text) => text.length > 200);
|
||||||
|
blocks.sort((a, b) => b.length - a.length);
|
||||||
|
|
||||||
|
return {
|
||||||
|
applyLinks,
|
||||||
|
description: blocks[0]?.slice(0, 5_000),
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
async function openCardAndExtractDetail(
|
||||||
|
page: Page,
|
||||||
|
selectors: readonly string[],
|
||||||
|
index: number,
|
||||||
|
docId: string | undefined,
|
||||||
|
): Promise<ScrapedDetail | null> {
|
||||||
|
const elementHandle = await page.evaluateHandle(
|
||||||
|
(args: { selectors: readonly string[]; idx: number }) => {
|
||||||
|
for (const selector of args.selectors) {
|
||||||
|
const items = document.querySelectorAll(selector);
|
||||||
|
if (items.length > 0) return items[args.idx] ?? null;
|
||||||
|
}
|
||||||
|
return null;
|
||||||
|
},
|
||||||
|
{ selectors, idx: index },
|
||||||
|
);
|
||||||
|
|
||||||
|
const element = elementHandle.asElement();
|
||||||
|
if (!element) return null;
|
||||||
|
|
||||||
|
await element.scrollIntoViewIfNeeded();
|
||||||
|
// A single click is unreliable here: clicking a card while a *different*
|
||||||
|
// card's detail pane is already open frequently just closes the open pane
|
||||||
|
// instead of switching to the new one (confirmed by watching real runs —
|
||||||
|
// it alternates open/closed rather than opening the newly clicked card).
|
||||||
|
// A double-click reopens reliably; verified across 6 sequential cards.
|
||||||
|
await element.dblclick({ timeout: 5_000, delay: 80 });
|
||||||
|
if (docId) {
|
||||||
|
try {
|
||||||
|
await page.waitForSelector(`[data-encoded-docid="${docId}"]`, {
|
||||||
|
timeout: 6_000,
|
||||||
|
});
|
||||||
|
} catch {
|
||||||
|
// fall through — extractDetailScript falls back to a document-wide
|
||||||
|
// scan for apply links if the docid-scoped block never shows up.
|
||||||
|
}
|
||||||
|
}
|
||||||
|
await page.waitForTimeout(500);
|
||||||
|
return page.evaluate(extractDetailScript, docId);
|
||||||
|
}
|
||||||
|
|
||||||
|
async function saveDebugArtifacts(page: Page): Promise<void> {
|
||||||
|
try {
|
||||||
|
await mkdir(dirname(DEBUG_HTML_PATH), { recursive: true });
|
||||||
|
await writeFile(DEBUG_HTML_PATH, await page.content(), "utf-8");
|
||||||
|
await page.screenshot({ path: DEBUG_SCREENSHOT_PATH, fullPage: true });
|
||||||
|
} catch (error) {
|
||||||
|
console.error(
|
||||||
|
`[google-jobs] failed to save debug artifacts: ${error instanceof Error ? error.message : String(error)}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function scrapeTerm(
|
||||||
|
context: BrowserContext,
|
||||||
|
term: string,
|
||||||
|
remainingCap: number,
|
||||||
|
options: { headless: boolean; unblockTimeoutMs: number },
|
||||||
|
): Promise<ScrapedJob[]> {
|
||||||
|
const page = await context.newPage();
|
||||||
|
const out: ScrapedJob[] = [];
|
||||||
|
|
||||||
|
try {
|
||||||
|
await page.goto(buildSearchUrl(term), {
|
||||||
|
waitUntil: "domcontentloaded",
|
||||||
|
timeout: 45_000,
|
||||||
|
});
|
||||||
|
await dismissConsent(page);
|
||||||
|
await page.waitForTimeout(1_500);
|
||||||
|
|
||||||
|
if (await isBlockedPage(page)) {
|
||||||
|
const solvedByHuman =
|
||||||
|
!options.headless &&
|
||||||
|
(await waitForHumanToUnblock(page, options.unblockTimeoutMs));
|
||||||
|
if (!solvedByHuman) {
|
||||||
|
throw new GoogleBlockedError(
|
||||||
|
"Google blocked this request (CAPTCHA / unusual traffic detection). This is common from shared, VPN, or datacenter IPs. Try again later, reduce run frequency, run from a residential network, set GOOGLE_JOBS_PROXY, or set GOOGLE_JOBS_HEADLESS=false to solve it yourself.",
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
const listReady = await waitForJobsList(page);
|
||||||
|
if (!listReady) {
|
||||||
|
if (await isEmptyJobsVertical(page)) {
|
||||||
|
console.error(
|
||||||
|
`[google-jobs] Google's Jobs tab reported no matches for "${term}" (not a block or selector issue — try a broader/more common phrasing).`,
|
||||||
|
);
|
||||||
|
} else {
|
||||||
|
console.error(
|
||||||
|
`[google-jobs] Jobs list did not appear for "${term}" (url: ${page.url()}). Saved a screenshot + HTML dump to ${DEBUG_SCREENSHOT_PATH} for inspection.`,
|
||||||
|
);
|
||||||
|
await saveDebugArtifacts(page);
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
await growJobsList(page, CARD_SELECTORS, remainingCap);
|
||||||
|
|
||||||
|
const cards = await page.evaluate(extractCardsScript, CARD_SELECTORS);
|
||||||
|
const limit = Math.min(cards.length, remainingCap);
|
||||||
|
|
||||||
|
for (let index = 0; index < limit; index += 1) {
|
||||||
|
const card = cards[index];
|
||||||
|
// Google's own permalink into this specific listing — always available
|
||||||
|
// from the list view, so results stay unique/useful even if the detail
|
||||||
|
// click below fails for this card.
|
||||||
|
const shareUrl = card.shareUrl;
|
||||||
|
try {
|
||||||
|
const detail = await openCardAndExtractDetail(
|
||||||
|
page,
|
||||||
|
CARD_SELECTORS,
|
||||||
|
index,
|
||||||
|
card.docId,
|
||||||
|
);
|
||||||
|
const applyLink = detail?.applyLinks[0];
|
||||||
|
const jobUrl = shareUrl ?? applyLink?.href ?? page.url();
|
||||||
|
|
||||||
|
out.push({
|
||||||
|
title: card.title,
|
||||||
|
employer: card.employer,
|
||||||
|
jobUrl,
|
||||||
|
applicationLink: applyLink?.href,
|
||||||
|
location: card.location,
|
||||||
|
jobDescription: detail?.description,
|
||||||
|
postedVia: card.via,
|
||||||
|
isRemote: /remote/i.test(`${card.location ?? ""} ${card.title}`),
|
||||||
|
});
|
||||||
|
} catch (error) {
|
||||||
|
console.error(
|
||||||
|
`[google-jobs] card ${index} for "${term}" failed: ${
|
||||||
|
error instanceof Error ? error.message : String(error)
|
||||||
|
}`,
|
||||||
|
);
|
||||||
|
if (shareUrl) {
|
||||||
|
// Detail click failed, but we still have a unique, valid link to
|
||||||
|
// the listing from the card itself — don't drop the job entirely.
|
||||||
|
out.push({
|
||||||
|
title: card.title,
|
||||||
|
employer: card.employer,
|
||||||
|
jobUrl: shareUrl,
|
||||||
|
location: card.location,
|
||||||
|
postedVia: card.via,
|
||||||
|
isRemote: /remote/i.test(`${card.location ?? ""} ${card.title}`),
|
||||||
|
});
|
||||||
|
}
|
||||||
|
}
|
||||||
|
await page.waitForTimeout(300 + Math.floor(Math.random() * 400));
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
await page.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
return out;
|
||||||
|
}
|
||||||
|
|
||||||
|
async function main(): Promise<void> {
|
||||||
|
const terms = parseTerms(process.env.GOOGLE_JOBS_SEARCH_TERMS);
|
||||||
|
const maxJobsEnv = Number.parseInt(
|
||||||
|
process.env.GOOGLE_JOBS_MAX_JOBS ?? "30",
|
||||||
|
10,
|
||||||
|
);
|
||||||
|
const cap = Number.isFinite(maxJobsEnv)
|
||||||
|
? Math.max(1, Math.min(maxJobsEnv, 150))
|
||||||
|
: 30;
|
||||||
|
// Headed mode lets a human solve a CAPTCHA in the visible window; only
|
||||||
|
// useful when run directly on a machine with a display, not on a headless
|
||||||
|
// cron/server pipeline.
|
||||||
|
const headless = process.env.GOOGLE_JOBS_HEADLESS !== "false";
|
||||||
|
const unblockTimeoutMs = Number.parseInt(
|
||||||
|
process.env.GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS ?? "300000",
|
||||||
|
10,
|
||||||
|
);
|
||||||
|
|
||||||
|
const browser = await launchBrowser(headless);
|
||||||
|
const hasSavedState = await access(STATE_PATH)
|
||||||
|
.then(() => true)
|
||||||
|
.catch(() => false);
|
||||||
|
const context = await browser.newContext(
|
||||||
|
hasSavedState ? { storageState: STATE_PATH } : {},
|
||||||
|
);
|
||||||
|
// tsx/esbuild compiles this file with `keepNames`, which injects
|
||||||
|
// `__name(fn, "fn")` calls into any named function it transpiles —
|
||||||
|
// including ones later serialized into `page.evaluate()` scripts via
|
||||||
|
// `Function.prototype.toString()`. That helper only exists in our Node
|
||||||
|
// process, not in the browser page, so without this no-op polyfill any
|
||||||
|
// evaluate script with a nested named function throws
|
||||||
|
// `ReferenceError: __name is not defined`.
|
||||||
|
await context.addInitScript(() => {
|
||||||
|
(window as unknown as { __name?: (fn: unknown) => unknown }).__name = (
|
||||||
|
fn: unknown,
|
||||||
|
) => fn;
|
||||||
|
});
|
||||||
|
|
||||||
|
const all: ScrapedJob[] = [];
|
||||||
|
const seenUrls = new Set<string>();
|
||||||
|
|
||||||
|
try {
|
||||||
|
for (const term of terms) {
|
||||||
|
if (all.length >= cap) break;
|
||||||
|
try {
|
||||||
|
const rows = await scrapeTerm(context, term, cap - all.length, {
|
||||||
|
headless,
|
||||||
|
unblockTimeoutMs,
|
||||||
|
});
|
||||||
|
for (const row of rows) {
|
||||||
|
if (all.length >= cap) break;
|
||||||
|
if (seenUrls.has(row.jobUrl)) continue;
|
||||||
|
seenUrls.add(row.jobUrl);
|
||||||
|
all.push(row);
|
||||||
|
}
|
||||||
|
} catch (error) {
|
||||||
|
if (error instanceof GoogleBlockedError) {
|
||||||
|
// IP-level block affects the whole run; stop trying further terms
|
||||||
|
// and let it propagate so the process exits non-zero with a clear message.
|
||||||
|
throw error;
|
||||||
|
}
|
||||||
|
console.error(
|
||||||
|
`[google-jobs] term "${term}" failed: ${
|
||||||
|
error instanceof Error ? error.message : String(error)
|
||||||
|
}`,
|
||||||
|
);
|
||||||
|
}
|
||||||
|
}
|
||||||
|
} finally {
|
||||||
|
// Persist cookies/session (even a partially solved one) so future
|
||||||
|
// headless/automated runs can reuse it and hit fewer CAPTCHA walls.
|
||||||
|
await mkdir(dirname(STATE_PATH), { recursive: true });
|
||||||
|
await context.storageState({ path: STATE_PATH }).catch(() => undefined);
|
||||||
|
await browser.close();
|
||||||
|
}
|
||||||
|
|
||||||
|
await mkdir(dirname(OUTPUT_PATH), { recursive: true });
|
||||||
|
await writeFile(OUTPUT_PATH, JSON.stringify(all, null, 2), "utf-8");
|
||||||
|
}
|
||||||
|
|
||||||
|
main().catch((error) => {
|
||||||
|
console.error(error instanceof Error ? error.message : String(error));
|
||||||
|
process.exit(1);
|
||||||
|
});
|
||||||
@@ -0,0 +1,154 @@
|
|||||||
|
import { spawn, spawnSync } from "node:child_process";
|
||||||
|
import { readFile } from "node:fs/promises";
|
||||||
|
import { createRequire } from "node:module";
|
||||||
|
import { dirname, join } from "node:path";
|
||||||
|
import { fileURLToPath } from "node:url";
|
||||||
|
import { envForExtractorSubprocess } from "@shared/extractor-subprocess-env.js";
|
||||||
|
import type { CreateJobInput } from "@shared/types/jobs";
|
||||||
|
|
||||||
|
const srcDir = dirname(fileURLToPath(import.meta.url));
|
||||||
|
const EXTRACTOR_DIR = join(srcDir, "..");
|
||||||
|
const OUTPUT_PATH = join(EXTRACTOR_DIR, "storage/jobs.json");
|
||||||
|
const require = createRequire(import.meta.url);
|
||||||
|
const TSX_CLI_PATH = (() => {
|
||||||
|
try {
|
||||||
|
return require.resolve("tsx/dist/cli.mjs");
|
||||||
|
} catch {
|
||||||
|
return null;
|
||||||
|
}
|
||||||
|
})();
|
||||||
|
|
||||||
|
/** Browser scrape can take a while (per-term navigation + per-card clicks). */
|
||||||
|
const SUBPROCESS_TIMEOUT_MS = 4 * 60 * 1000;
|
||||||
|
|
||||||
|
interface ScrapedJob {
|
||||||
|
title?: string;
|
||||||
|
employer?: string;
|
||||||
|
jobUrl?: string;
|
||||||
|
applicationLink?: string;
|
||||||
|
location?: string;
|
||||||
|
jobDescription?: string;
|
||||||
|
postedVia?: string;
|
||||||
|
isRemote?: boolean;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface RunGoogleJobsOptions {
|
||||||
|
searchTerms?: string[];
|
||||||
|
maxJobs?: number;
|
||||||
|
}
|
||||||
|
|
||||||
|
export interface GoogleJobsResult {
|
||||||
|
success: boolean;
|
||||||
|
jobs: CreateJobInput[];
|
||||||
|
error?: string;
|
||||||
|
}
|
||||||
|
|
||||||
|
function canRunNpmCommand(): boolean {
|
||||||
|
const result = spawnSync("npm", ["--version"], { stdio: "ignore" });
|
||||||
|
return !result.error && result.status === 0;
|
||||||
|
}
|
||||||
|
|
||||||
|
function mapJob(row: ScrapedJob): CreateJobInput | null {
|
||||||
|
const jobUrl = row.jobUrl?.trim();
|
||||||
|
if (!jobUrl) return null;
|
||||||
|
return {
|
||||||
|
source: "google-jobs",
|
||||||
|
title: row.title?.trim() || "Unknown Title",
|
||||||
|
employer: row.employer?.trim() || "Unknown Employer",
|
||||||
|
jobUrl,
|
||||||
|
applicationLink: row.applicationLink?.trim() || jobUrl,
|
||||||
|
location: row.location?.trim() || undefined,
|
||||||
|
jobDescription: row.jobDescription?.trim() || undefined,
|
||||||
|
isRemote: row.isRemote ?? undefined,
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
export async function runGoogleJobs(
|
||||||
|
options: RunGoogleJobsOptions = {},
|
||||||
|
): Promise<GoogleJobsResult> {
|
||||||
|
const searchTerms =
|
||||||
|
options.searchTerms && options.searchTerms.length > 0
|
||||||
|
? options.searchTerms
|
||||||
|
: ["software engineer"];
|
||||||
|
const maxJobs = options.maxJobs ?? 30;
|
||||||
|
|
||||||
|
const useNpmCommand = canRunNpmCommand();
|
||||||
|
if (!TSX_CLI_PATH && !useNpmCommand) {
|
||||||
|
return {
|
||||||
|
success: false,
|
||||||
|
jobs: [],
|
||||||
|
error: "Unable to execute Google Jobs extractor (npm/tsx unavailable)",
|
||||||
|
};
|
||||||
|
}
|
||||||
|
|
||||||
|
try {
|
||||||
|
await new Promise<void>((resolve, reject) => {
|
||||||
|
const extractorEnv = envForExtractorSubprocess({
|
||||||
|
...process.env,
|
||||||
|
GOOGLE_JOBS_SEARCH_TERMS: JSON.stringify(searchTerms),
|
||||||
|
GOOGLE_JOBS_MAX_JOBS: String(maxJobs),
|
||||||
|
});
|
||||||
|
|
||||||
|
const child = TSX_CLI_PATH
|
||||||
|
? spawn(process.execPath, [TSX_CLI_PATH, join(srcDir, "main.ts")], {
|
||||||
|
cwd: EXTRACTOR_DIR,
|
||||||
|
env: extractorEnv,
|
||||||
|
stdio: ["ignore", "pipe", "pipe"],
|
||||||
|
})
|
||||||
|
: spawn("npx", ["tsx", join(srcDir, "main.ts")], {
|
||||||
|
cwd: EXTRACTOR_DIR,
|
||||||
|
env: extractorEnv,
|
||||||
|
stdio: ["ignore", "pipe", "pipe"],
|
||||||
|
});
|
||||||
|
|
||||||
|
let stderr = "";
|
||||||
|
child.stderr?.on("data", (chunk: Buffer) => {
|
||||||
|
stderr += chunk.toString();
|
||||||
|
});
|
||||||
|
|
||||||
|
const timeout = setTimeout(() => {
|
||||||
|
child.kill("SIGKILL");
|
||||||
|
reject(new Error("Google Jobs browser scrape timed out"));
|
||||||
|
}, SUBPROCESS_TIMEOUT_MS);
|
||||||
|
|
||||||
|
child.on("error", (error) => {
|
||||||
|
clearTimeout(timeout);
|
||||||
|
reject(error);
|
||||||
|
});
|
||||||
|
child.on("close", (code) => {
|
||||||
|
clearTimeout(timeout);
|
||||||
|
if (code === 0) {
|
||||||
|
resolve();
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
reject(
|
||||||
|
new Error(
|
||||||
|
stderr.trim() ||
|
||||||
|
`Google Jobs browser scrape exited with code ${code}`,
|
||||||
|
),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
});
|
||||||
|
|
||||||
|
const raw = await readFile(OUTPUT_PATH, "utf-8");
|
||||||
|
const parsed = JSON.parse(raw) as unknown;
|
||||||
|
const rows = Array.isArray(parsed) ? parsed : [];
|
||||||
|
|
||||||
|
const jobs: CreateJobInput[] = [];
|
||||||
|
const seen = new Set<string>();
|
||||||
|
for (const value of rows) {
|
||||||
|
if (!value || typeof value !== "object" || Array.isArray(value)) continue;
|
||||||
|
const mapped = mapJob(value as ScrapedJob);
|
||||||
|
if (!mapped) continue;
|
||||||
|
const key = mapped.jobUrl;
|
||||||
|
if (seen.has(key)) continue;
|
||||||
|
seen.add(key);
|
||||||
|
jobs.push(mapped);
|
||||||
|
}
|
||||||
|
|
||||||
|
return { success: true, jobs };
|
||||||
|
} catch (error) {
|
||||||
|
const message = error instanceof Error ? error.message : "Unknown error";
|
||||||
|
return { success: false, jobs: [], error: message };
|
||||||
|
}
|
||||||
|
}
|
||||||
@@ -0,0 +1,17 @@
|
|||||||
|
{
|
||||||
|
"compilerOptions": {
|
||||||
|
"module": "ESNext",
|
||||||
|
"moduleResolution": "bundler",
|
||||||
|
"target": "ES2022",
|
||||||
|
"outDir": "dist",
|
||||||
|
"strict": true,
|
||||||
|
"noUnusedLocals": false,
|
||||||
|
"lib": ["ES2022", "DOM"],
|
||||||
|
"types": ["node"],
|
||||||
|
"baseUrl": ".",
|
||||||
|
"paths": {
|
||||||
|
"@shared/*": ["../../shared/src/*"]
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"include": ["./manifest.ts", "./src/**/*"]
|
||||||
|
}
|
||||||
@@ -116,11 +116,32 @@ router.addHandler(
|
|||||||
const title = (await titleLocator.textContent())?.trim() ?? null;
|
const title = (await titleLocator.textContent())?.trim() ?? null;
|
||||||
const jobUrl = toAbsolute(await titleLocator.getAttribute("href"));
|
const jobUrl = toAbsolute(await titleLocator.getAttribute("href"));
|
||||||
|
|
||||||
const employerImg = article.locator("figure img");
|
// Employer logo/link are optional — some cards omit <figure img>, and a
|
||||||
const employer = (await employerImg.getAttribute("alt"))?.trim() ?? null;
|
// hard getAttribute timeout used to fail the whole region page (0 jobs)
|
||||||
|
// even after we had already counted cards.
|
||||||
const employerAnchor = article.locator("figure a");
|
let employer: string | null = null;
|
||||||
const employerUrl = toAbsolute(await employerAnchor.getAttribute("href"));
|
let employerUrl: string | null = null;
|
||||||
|
try {
|
||||||
|
const employerImg = article.locator("figure img").first();
|
||||||
|
if ((await employerImg.count()) > 0) {
|
||||||
|
employer =
|
||||||
|
(
|
||||||
|
await employerImg.getAttribute("alt", { timeout: 2000 })
|
||||||
|
)?.trim() ?? null;
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// optional
|
||||||
|
}
|
||||||
|
try {
|
||||||
|
const employerAnchor = article.locator("figure a").first();
|
||||||
|
if ((await employerAnchor.count()) > 0) {
|
||||||
|
employerUrl = toAbsolute(
|
||||||
|
await employerAnchor.getAttribute("href", { timeout: 2000 }),
|
||||||
|
);
|
||||||
|
}
|
||||||
|
} catch {
|
||||||
|
// optional
|
||||||
|
}
|
||||||
|
|
||||||
let disciplines: string | null = null;
|
let disciplines: string | null = null;
|
||||||
try {
|
try {
|
||||||
|
|||||||
@@ -14,8 +14,10 @@ import type {
|
|||||||
import type { CreateJobInput } from "@shared/types/jobs";
|
import type { CreateJobInput } from "@shared/types/jobs";
|
||||||
|
|
||||||
const API_URL = "https://himalayas.app/jobs/api";
|
const API_URL = "https://himalayas.app/jobs/api";
|
||||||
const PAGE_SIZE = 50;
|
/** Himalayas silently caps `limit` at 20; requesting more makes us think the
|
||||||
const MAX_PAGES = 5;
|
* feed ended early (`raw.length < PAGE_SIZE`) and stop after the first page. */
|
||||||
|
const PAGE_SIZE = 20;
|
||||||
|
const MAX_PAGES = 10;
|
||||||
|
|
||||||
interface HimalayasJob {
|
interface HimalayasJob {
|
||||||
title?: string;
|
title?: string;
|
||||||
|
|||||||
@@ -1,3 +1,8 @@
|
|||||||
# python-jobspy requires Python 3.10+ (wheels not published for 3.9 and below).
|
# python-jobspy requires Python 3.10+ (wheels not published for 3.9 and below).
|
||||||
python-jobspy
|
#
|
||||||
|
# Pinned to 1.1.82: scrape_jobs.py monkeypatches Glassdoor._fetch_jobs_page
|
||||||
|
# to work around an upstream bug where any partial GraphQL error (even for
|
||||||
|
# unrelated fields) discards a whole page of valid job data. If bumping this
|
||||||
|
# version, re-check that method still matches before removing the pin.
|
||||||
|
python-jobspy==1.1.82
|
||||||
pandas
|
pandas
|
||||||
|
|||||||
@@ -1,12 +1,81 @@
|
|||||||
import csv
|
import csv
|
||||||
import json
|
import json
|
||||||
|
import logging
|
||||||
import os
|
import os
|
||||||
|
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||||
from pathlib import Path
|
from pathlib import Path
|
||||||
|
|
||||||
import pandas as pd
|
import pandas as pd
|
||||||
from jobspy import scrape_jobs
|
from jobspy import scrape_jobs
|
||||||
|
from jobspy.exception import GlassdoorException
|
||||||
|
from jobspy.glassdoor import Glassdoor
|
||||||
|
from jobspy.glassdoor.util import get_cursor_for_page
|
||||||
|
|
||||||
PROGRESS_PREFIX = "JOBOPS_PROGRESS "
|
PROGRESS_PREFIX = "JOBOPS_PROGRESS "
|
||||||
|
_glassdoor_log = logging.getLogger("JobSpy:Glassdoor")
|
||||||
|
|
||||||
|
|
||||||
|
def _patched_glassdoor_fetch_jobs_page(
|
||||||
|
self, scraper_input, location_id, location_type, page_num, cursor
|
||||||
|
):
|
||||||
|
"""Replaces python-jobspy's Glassdoor._fetch_jobs_page (as of 1.1.82).
|
||||||
|
|
||||||
|
Upstream treats *any* `errors` entry in Glassdoor's GraphQL response as
|
||||||
|
fatal and discards the whole page, even when the job listings we need
|
||||||
|
parsed fine. In practice Glassdoor's backend intermittently (but
|
||||||
|
consistently, as of writing) returns a partial error for an unrelated
|
||||||
|
field — observed: `jobsPageSeoData` failing with
|
||||||
|
`SUBREQUEST_HTTP_ERROR` / "dns error" on Glassdoor's own infra — while
|
||||||
|
`data.jobListings.jobListings` still comes back with real results. The
|
||||||
|
upstream check throws that good data away, so every Glassdoor run
|
||||||
|
returns 0 jobs. This patch only bails out when the job listings payload
|
||||||
|
itself is actually missing.
|
||||||
|
"""
|
||||||
|
jobs = []
|
||||||
|
self.scraper_input = scraper_input
|
||||||
|
try:
|
||||||
|
payload = self._add_payload(location_id, location_type, page_num, cursor)
|
||||||
|
response = self.session.post(
|
||||||
|
f"{self.base_url}/graph",
|
||||||
|
timeout_seconds=15,
|
||||||
|
data=payload,
|
||||||
|
)
|
||||||
|
if response.status_code != 200:
|
||||||
|
raise GlassdoorException(
|
||||||
|
f"bad response status code: {response.status_code}"
|
||||||
|
)
|
||||||
|
res_json = response.json()[0]
|
||||||
|
job_listings = res_json.get("data", {}).get("jobListings")
|
||||||
|
if job_listings is None:
|
||||||
|
raise ValueError("Error encountered in API response")
|
||||||
|
if "errors" in res_json:
|
||||||
|
_glassdoor_log.warning(
|
||||||
|
"Glassdoor GraphQL returned partial errors for unrelated "
|
||||||
|
"fields; continuing with the job listings that did parse: %s",
|
||||||
|
res_json["errors"],
|
||||||
|
)
|
||||||
|
except (GlassdoorException, ValueError, Exception) as e:
|
||||||
|
_glassdoor_log.error(f"Glassdoor: {str(e)}")
|
||||||
|
return jobs, None
|
||||||
|
|
||||||
|
jobs_data = job_listings["jobListings"]
|
||||||
|
|
||||||
|
with ThreadPoolExecutor(max_workers=self.jobs_per_page) as executor:
|
||||||
|
future_to_job_data = {
|
||||||
|
executor.submit(self._process_job, job): job for job in jobs_data
|
||||||
|
}
|
||||||
|
for future in as_completed(future_to_job_data):
|
||||||
|
try:
|
||||||
|
job_post = future.result()
|
||||||
|
if job_post:
|
||||||
|
jobs.append(job_post)
|
||||||
|
except Exception as exc:
|
||||||
|
raise GlassdoorException(f"Glassdoor generated an exception: {exc}")
|
||||||
|
|
||||||
|
return jobs, get_cursor_for_page(job_listings["paginationCursors"], page_num + 1)
|
||||||
|
|
||||||
|
|
||||||
|
Glassdoor._fetch_jobs_page = _patched_glassdoor_fetch_jobs_page
|
||||||
COUNTRY_ALIASES = {
|
COUNTRY_ALIASES = {
|
||||||
"uk": "united kingdom",
|
"uk": "united kingdom",
|
||||||
"united kingdom": "united kingdom",
|
"united kingdom": "united kingdom",
|
||||||
|
|||||||
@@ -394,15 +394,15 @@ export async function activateSearchProfileForBasicAuthUser(
|
|||||||
}
|
}
|
||||||
if (!listPayload.ok || !Array.isArray(listPayload.data)) return;
|
if (!listPayload.ok || !Array.isArray(listPayload.data)) return;
|
||||||
const normalizedUsername = username.trim().toLowerCase();
|
const normalizedUsername = username.trim().toLowerCase();
|
||||||
const match = listPayload.data.find((row) => {
|
const matches = listPayload.data.filter((row) => {
|
||||||
const aliases = (row.data?.basicAuthUser ?? "")
|
const aliases = (row.data?.basicAuthUser ?? "")
|
||||||
.split(",")
|
.split(",")
|
||||||
.map((part) => part.trim().toLowerCase())
|
.map((part) => part.trim().toLowerCase())
|
||||||
.filter(Boolean);
|
.filter(Boolean);
|
||||||
return aliases.includes(normalizedUsername);
|
return aliases.includes(normalizedUsername);
|
||||||
});
|
});
|
||||||
if (!match) return;
|
if (matches.length !== 1) return;
|
||||||
await fetch(`${API_BASE}/profiles/${match.id}/activate`, {
|
await fetch(`${API_BASE}/profiles/${matches[0].id}/activate`, {
|
||||||
method: "POST",
|
method: "POST",
|
||||||
headers: {
|
headers: {
|
||||||
"Content-Type": "application/json",
|
"Content-Type": "application/json",
|
||||||
|
|||||||
@@ -1,8 +1,17 @@
|
|||||||
|
import { PRIOR_SKIP_DISMISS_LOOKBACK_MS } from "@shared/job-fingerprint";
|
||||||
import { createJob } from "@shared/testing/factories";
|
import { createJob } from "@shared/testing/factories";
|
||||||
import { describe, expect, it } from "vitest";
|
import { describe, expect, it } from "vitest";
|
||||||
import { buildDuplicateDismissHints } from "./job-dedup";
|
import { buildDuplicateDismissHints } from "./job-dedup";
|
||||||
|
|
||||||
describe("buildDuplicateDismissHints", () => {
|
describe("buildDuplicateDismissHints", () => {
|
||||||
|
const now = Date.parse("2026-06-10T12:00:00.000Z");
|
||||||
|
const recentUpdatedAt = new Date(
|
||||||
|
now - PRIOR_SKIP_DISMISS_LOOKBACK_MS + 86_400_000,
|
||||||
|
).toISOString();
|
||||||
|
const staleUpdatedAt = new Date(
|
||||||
|
now - PRIOR_SKIP_DISMISS_LOOKBACK_MS - 86_400_000,
|
||||||
|
).toISOString();
|
||||||
|
|
||||||
it("flags open jobs that match a skipped posting", () => {
|
it("flags open jobs that match a skipped posting", () => {
|
||||||
const jobs = [
|
const jobs = [
|
||||||
createJob({
|
createJob({
|
||||||
@@ -10,6 +19,7 @@ describe("buildDuplicateDismissHints", () => {
|
|||||||
employer: "Acme",
|
employer: "Acme",
|
||||||
title: "SDET",
|
title: "SDET",
|
||||||
status: "skipped",
|
status: "skipped",
|
||||||
|
updatedAt: recentUpdatedAt,
|
||||||
}),
|
}),
|
||||||
createJob({
|
createJob({
|
||||||
id: "open-1",
|
id: "open-1",
|
||||||
@@ -25,8 +35,71 @@ describe("buildDuplicateDismissHints", () => {
|
|||||||
}),
|
}),
|
||||||
];
|
];
|
||||||
|
|
||||||
const hints = buildDuplicateDismissHints(jobs);
|
const hints = buildDuplicateDismissHints(jobs, now);
|
||||||
expect(hints.get("open-1")).toBe("skipped");
|
expect(hints.get("open-1")).toBe("skipped");
|
||||||
expect(hints.has("open-2")).toBe(false);
|
expect(hints.has("open-2")).toBe(false);
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("matches employer prefix and plural title variants", () => {
|
||||||
|
const jobs = [
|
||||||
|
createJob({
|
||||||
|
id: "skipped-1",
|
||||||
|
employer: "CGI",
|
||||||
|
title: "Automation Test Engineer",
|
||||||
|
status: "skipped",
|
||||||
|
updatedAt: recentUpdatedAt,
|
||||||
|
}),
|
||||||
|
createJob({
|
||||||
|
id: "open-1",
|
||||||
|
employer: ". CGI IT UK Limited",
|
||||||
|
title: "Automation Test Engineers",
|
||||||
|
status: "discovered",
|
||||||
|
}),
|
||||||
|
];
|
||||||
|
|
||||||
|
const hints = buildDuplicateDismissHints(jobs, now);
|
||||||
|
expect(hints.get("open-1")).toBe("skipped");
|
||||||
|
});
|
||||||
|
|
||||||
|
it("ignores skipped jobs older than the lookback window", () => {
|
||||||
|
const jobs = [
|
||||||
|
createJob({
|
||||||
|
id: "skipped-1",
|
||||||
|
employer: "Acme",
|
||||||
|
title: "SDET",
|
||||||
|
status: "skipped",
|
||||||
|
updatedAt: staleUpdatedAt,
|
||||||
|
}),
|
||||||
|
createJob({
|
||||||
|
id: "open-1",
|
||||||
|
employer: "Acme",
|
||||||
|
title: "SDET",
|
||||||
|
status: "discovered",
|
||||||
|
}),
|
||||||
|
];
|
||||||
|
|
||||||
|
const hints = buildDuplicateDismissHints(jobs, now);
|
||||||
|
expect(hints.has("open-1")).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("does not match different titles at the same employer", () => {
|
||||||
|
const jobs = [
|
||||||
|
createJob({
|
||||||
|
id: "skipped-1",
|
||||||
|
employer: "Cognizant",
|
||||||
|
title: "Automation Test Engineer",
|
||||||
|
status: "skipped",
|
||||||
|
updatedAt: recentUpdatedAt,
|
||||||
|
}),
|
||||||
|
createJob({
|
||||||
|
id: "open-1",
|
||||||
|
employer: "Cognizant",
|
||||||
|
title: "AI Automation Test Engineer",
|
||||||
|
status: "discovered",
|
||||||
|
}),
|
||||||
|
];
|
||||||
|
|
||||||
|
const hints = buildDuplicateDismissHints(jobs, now);
|
||||||
|
expect(hints.has("open-1")).toBe(false);
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -1,44 +1,60 @@
|
|||||||
import { collectJobDedupKeys } from "@shared/job-fingerprint";
|
import {
|
||||||
|
jobsMatchEmployerAndTitleDismiss,
|
||||||
|
PRIOR_SKIP_DISMISS_LOOKBACK_MS,
|
||||||
|
} from "@shared/job-fingerprint";
|
||||||
import type { JobListItem, JobStatus } from "@shared/types";
|
import type { JobListItem, JobStatus } from "@shared/types";
|
||||||
|
|
||||||
export type DuplicateDismissReason = "skipped" | "applied";
|
export type DuplicateDismissReason = "skipped" | "applied";
|
||||||
|
|
||||||
/**
|
const OPEN_JOB_STATUSES = new Set<JobStatus>([
|
||||||
* Map open jobs to a prior skip/apply when employer+title or description matches.
|
|
||||||
*/
|
|
||||||
export function buildDuplicateDismissHints(
|
|
||||||
jobs: readonly JobListItem[],
|
|
||||||
): Map<string, DuplicateDismissReason> {
|
|
||||||
const dismissedKeys = new Map<string, DuplicateDismissReason>();
|
|
||||||
|
|
||||||
for (const job of jobs) {
|
|
||||||
if (job.status !== "skipped" && job.status !== "applied") continue;
|
|
||||||
const reason: DuplicateDismissReason =
|
|
||||||
job.status === "applied" ? "applied" : "skipped";
|
|
||||||
for (const key of collectJobDedupKeys({
|
|
||||||
employer: job.employer,
|
|
||||||
title: job.title,
|
|
||||||
})) {
|
|
||||||
if (!dismissedKeys.has(key)) dismissedKeys.set(key, reason);
|
|
||||||
}
|
|
||||||
}
|
|
||||||
|
|
||||||
const hints = new Map<string, DuplicateDismissReason>();
|
|
||||||
const openStatuses = new Set<JobStatus>([
|
|
||||||
"discovered",
|
"discovered",
|
||||||
"ready",
|
"ready",
|
||||||
"processing",
|
"processing",
|
||||||
]);
|
]);
|
||||||
|
|
||||||
|
function isWithinDismissLookback(updatedAt: string, nowMs: number): boolean {
|
||||||
|
const updatedMs = Date.parse(updatedAt);
|
||||||
|
if (!Number.isFinite(updatedMs)) return false;
|
||||||
|
return nowMs - updatedMs <= PRIOR_SKIP_DISMISS_LOOKBACK_MS;
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Map open jobs to a prior skip/apply when the same company and title match
|
||||||
|
* within the last 90 days (see {@link PRIOR_SKIP_DISMISS_LOOKBACK_MS}).
|
||||||
|
*/
|
||||||
|
export function buildDuplicateDismissHints(
|
||||||
|
jobs: readonly JobListItem[],
|
||||||
|
nowMs: number = Date.now(),
|
||||||
|
): Map<string, DuplicateDismissReason> {
|
||||||
|
const dismissed: Array<{
|
||||||
|
employer: string;
|
||||||
|
title: string;
|
||||||
|
reason: DuplicateDismissReason;
|
||||||
|
}> = [];
|
||||||
|
|
||||||
for (const job of jobs) {
|
for (const job of jobs) {
|
||||||
if (!openStatuses.has(job.status)) continue;
|
if (job.status !== "skipped" && job.status !== "applied") continue;
|
||||||
for (const key of collectJobDedupKeys({
|
if (!isWithinDismissLookback(job.updatedAt, nowMs)) continue;
|
||||||
|
dismissed.push({
|
||||||
employer: job.employer,
|
employer: job.employer,
|
||||||
title: job.title,
|
title: job.title,
|
||||||
})) {
|
reason: job.status === "applied" ? "applied" : "skipped",
|
||||||
const reason = dismissedKeys.get(key);
|
});
|
||||||
if (reason) {
|
}
|
||||||
hints.set(job.id, reason);
|
|
||||||
|
const hints = new Map<string, DuplicateDismissReason>();
|
||||||
|
for (const job of jobs) {
|
||||||
|
if (!OPEN_JOB_STATUSES.has(job.status)) continue;
|
||||||
|
for (const prior of dismissed) {
|
||||||
|
if (
|
||||||
|
jobsMatchEmployerAndTitleDismiss({
|
||||||
|
employerA: job.employer,
|
||||||
|
titleA: job.title,
|
||||||
|
employerB: prior.employer,
|
||||||
|
titleB: prior.title,
|
||||||
|
})
|
||||||
|
) {
|
||||||
|
hints.set(job.id, prior.reason);
|
||||||
break;
|
break;
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
@@ -46,5 +62,3 @@ export function buildDuplicateDismissHints(
|
|||||||
|
|
||||||
return hints;
|
return hints;
|
||||||
}
|
}
|
||||||
|
|
||||||
export { collectJobDedupKeys };
|
|
||||||
|
|||||||
@@ -63,6 +63,8 @@ export const OrchestratorPage: React.FC = () => {
|
|||||||
setEmployerFilterTokens,
|
setEmployerFilterTokens,
|
||||||
applySettingsCompanySkipList,
|
applySettingsCompanySkipList,
|
||||||
setApplySettingsCompanySkipList,
|
setApplySettingsCompanySkipList,
|
||||||
|
hidePriorSkips,
|
||||||
|
setHidePriorSkips,
|
||||||
sort,
|
sort,
|
||||||
setSort,
|
setSort,
|
||||||
resetFilters,
|
resetFilters,
|
||||||
@@ -173,8 +175,8 @@ export const OrchestratorPage: React.FC = () => {
|
|||||||
);
|
);
|
||||||
|
|
||||||
const duplicateDismissHints = useMemo(
|
const duplicateDismissHints = useMemo(
|
||||||
() => buildDuplicateDismissHints(jobs),
|
() => (hidePriorSkips ? buildDuplicateDismissHints(jobs) : undefined),
|
||||||
[jobs],
|
[hidePriorSkips, jobs],
|
||||||
);
|
);
|
||||||
|
|
||||||
const jobListFilterExtras = useMemo(
|
const jobListFilterExtras = useMemo(
|
||||||
@@ -531,6 +533,8 @@ export const OrchestratorPage: React.FC = () => {
|
|||||||
onApplySettingsCompanySkipListChange={
|
onApplySettingsCompanySkipListChange={
|
||||||
setApplySettingsCompanySkipList
|
setApplySettingsCompanySkipList
|
||||||
}
|
}
|
||||||
|
hidePriorSkips={hidePriorSkips}
|
||||||
|
onHidePriorSkipsChange={setHidePriorSkips}
|
||||||
sponsorFilter={sponsorFilter}
|
sponsorFilter={sponsorFilter}
|
||||||
onSponsorFilterChange={setSponsorFilter}
|
onSponsorFilterChange={setSponsorFilter}
|
||||||
sponsorshipSignalsFilter={sponsorshipSignalsFilter}
|
sponsorshipSignalsFilter={sponsorshipSignalsFilter}
|
||||||
|
|||||||
@@ -640,6 +640,7 @@ export const AutomaticRunTab: React.FC<AutomaticRunTabProps> = ({
|
|||||||
</CardHeader>
|
</CardHeader>
|
||||||
<CardContent className="space-y-4">
|
<CardContent className="space-y-4">
|
||||||
<KeywordSetPicker
|
<KeywordSetPicker
|
||||||
|
key={settings?.activeProfileId ?? "default"}
|
||||||
disabled={isSaving || isPipelineRunning}
|
disabled={isSaving || isPipelineRunning}
|
||||||
onActiveSetChange={handleActiveKeywordSetChange}
|
onActiveSetChange={handleActiveKeywordSetChange}
|
||||||
/>
|
/>
|
||||||
|
|||||||
@@ -53,6 +53,8 @@ const renderFilters = (overrides?: Partial<FiltersProps>) => {
|
|||||||
onEmployerIncludeChange: vi.fn(),
|
onEmployerIncludeChange: vi.fn(),
|
||||||
onEmployerExcludeChange: vi.fn(),
|
onEmployerExcludeChange: vi.fn(),
|
||||||
applySettingsCompanySkipList: true,
|
applySettingsCompanySkipList: true,
|
||||||
|
hidePriorSkips: true,
|
||||||
|
onHidePriorSkipsChange: vi.fn(),
|
||||||
onApplySettingsCompanySkipListChange: vi.fn(),
|
onApplySettingsCompanySkipListChange: vi.fn(),
|
||||||
sponsorFilter: "all" as SponsorFilter,
|
sponsorFilter: "all" as SponsorFilter,
|
||||||
onSponsorFilterChange: vi.fn(),
|
onSponsorFilterChange: vi.fn(),
|
||||||
|
|||||||
@@ -69,6 +69,8 @@ interface OrchestratorFiltersProps {
|
|||||||
onEmployerExcludeChange: (values: string[]) => void;
|
onEmployerExcludeChange: (values: string[]) => void;
|
||||||
applySettingsCompanySkipList: boolean;
|
applySettingsCompanySkipList: boolean;
|
||||||
onApplySettingsCompanySkipListChange: (value: boolean) => void;
|
onApplySettingsCompanySkipListChange: (value: boolean) => void;
|
||||||
|
hidePriorSkips: boolean;
|
||||||
|
onHidePriorSkipsChange: (value: boolean) => void;
|
||||||
sponsorFilter: SponsorFilter;
|
sponsorFilter: SponsorFilter;
|
||||||
onSponsorFilterChange: (value: SponsorFilter) => void;
|
onSponsorFilterChange: (value: SponsorFilter) => void;
|
||||||
sponsorshipSignalsFilter: SponsorshipSignalId[];
|
sponsorshipSignalsFilter: SponsorshipSignalId[];
|
||||||
@@ -183,6 +185,8 @@ export const OrchestratorFilters: React.FC<OrchestratorFiltersProps> = ({
|
|||||||
onEmployerExcludeChange,
|
onEmployerExcludeChange,
|
||||||
applySettingsCompanySkipList,
|
applySettingsCompanySkipList,
|
||||||
onApplySettingsCompanySkipListChange,
|
onApplySettingsCompanySkipListChange,
|
||||||
|
hidePriorSkips,
|
||||||
|
onHidePriorSkipsChange,
|
||||||
sponsorFilter,
|
sponsorFilter,
|
||||||
onSponsorFilterChange,
|
onSponsorFilterChange,
|
||||||
sponsorshipSignalsFilter,
|
sponsorshipSignalsFilter,
|
||||||
@@ -253,6 +257,7 @@ export const OrchestratorFilters: React.FC<OrchestratorFiltersProps> = ({
|
|||||||
employerIncludeFilter.length > 0 || employerExcludeFilter.length > 0,
|
employerIncludeFilter.length > 0 || employerExcludeFilter.length > 0,
|
||||||
) +
|
) +
|
||||||
Number(!applySettingsCompanySkipList) +
|
Number(!applySettingsCompanySkipList) +
|
||||||
|
Number(!hidePriorSkips) +
|
||||||
Number(sponsorFilter !== "all") +
|
Number(sponsorFilter !== "all") +
|
||||||
Number(sponsorshipSignalsFilter.length > 0) +
|
Number(sponsorshipSignalsFilter.length > 0) +
|
||||||
Number(hideSponsorBlockers) +
|
Number(hideSponsorBlockers) +
|
||||||
@@ -271,6 +276,7 @@ export const OrchestratorFilters: React.FC<OrchestratorFiltersProps> = ({
|
|||||||
employerIncludeFilter.length,
|
employerIncludeFilter.length,
|
||||||
employerExcludeFilter.length,
|
employerExcludeFilter.length,
|
||||||
applySettingsCompanySkipList,
|
applySettingsCompanySkipList,
|
||||||
|
hidePriorSkips,
|
||||||
sponsorFilter,
|
sponsorFilter,
|
||||||
sponsorshipSignalsFilter.length,
|
sponsorshipSignalsFilter.length,
|
||||||
hideSponsorBlockers,
|
hideSponsorBlockers,
|
||||||
@@ -542,6 +548,29 @@ export const OrchestratorFilters: React.FC<OrchestratorFiltersProps> = ({
|
|||||||
</div>
|
</div>
|
||||||
</div>
|
</div>
|
||||||
|
|
||||||
|
<div className="flex items-start gap-3 rounded-md border border-border/60 bg-muted/20 px-3 py-3">
|
||||||
|
<Checkbox
|
||||||
|
id="hide-prior-skips"
|
||||||
|
checked={hidePriorSkips}
|
||||||
|
onCheckedChange={(checked) => {
|
||||||
|
onHidePriorSkipsChange(checked === true);
|
||||||
|
}}
|
||||||
|
/>
|
||||||
|
<div className="space-y-1">
|
||||||
|
<label
|
||||||
|
htmlFor="hide-prior-skips"
|
||||||
|
className="text-sm font-medium leading-none cursor-pointer"
|
||||||
|
>
|
||||||
|
Hide roles you already skipped
|
||||||
|
</label>
|
||||||
|
<p className="text-xs text-muted-foreground">
|
||||||
|
Hides Discovered and Ready rows that match a skipped
|
||||||
|
or applied job with the same company and title from
|
||||||
|
the last 90 days.
|
||||||
|
</p>
|
||||||
|
</div>
|
||||||
|
</div>
|
||||||
|
|
||||||
<TokenizedInput
|
<TokenizedInput
|
||||||
id="employer-include-filter"
|
id="employer-include-filter"
|
||||||
values={employerIncludeFilter}
|
values={employerIncludeFilter}
|
||||||
|
|||||||
@@ -5,7 +5,7 @@ import type { SearchProfile } from "@shared/types";
|
|||||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||||
import { Loader2 } from "lucide-react";
|
import { Loader2 } from "lucide-react";
|
||||||
import type React from "react";
|
import type React from "react";
|
||||||
import { useMemo } from "react";
|
import { useEffect, useMemo, useState } from "react";
|
||||||
import { toast } from "sonner";
|
import { toast } from "sonner";
|
||||||
import { Button } from "@/components/ui/button";
|
import { Button } from "@/components/ui/button";
|
||||||
import {
|
import {
|
||||||
@@ -27,17 +27,38 @@ export const ProfileQuickSwitch: React.FC = () => {
|
|||||||
const { data: profiles = [], isLoading } = useQuery<SearchProfile[]>({
|
const { data: profiles = [], isLoading } = useQuery<SearchProfile[]>({
|
||||||
queryKey: queryKeys.searchProfiles.list(),
|
queryKey: queryKeys.searchProfiles.list(),
|
||||||
queryFn: api.listProfiles,
|
queryFn: api.listProfiles,
|
||||||
|
select: (rows) => {
|
||||||
|
const seen = new Set<string>();
|
||||||
|
return rows.filter((profile) => {
|
||||||
|
if (seen.has(profile.id)) return false;
|
||||||
|
seen.add(profile.id);
|
||||||
|
return true;
|
||||||
|
});
|
||||||
|
},
|
||||||
});
|
});
|
||||||
|
|
||||||
|
const [selectedId, setSelectedId] = useState<string | null>(null);
|
||||||
|
|
||||||
const activeId = useMemo(() => {
|
const activeId = useMemo(() => {
|
||||||
|
if (selectedId && profiles.some((profile) => profile.id === selectedId)) {
|
||||||
|
return selectedId;
|
||||||
|
}
|
||||||
const sid = settings?.activeProfileId?.trim() ?? "";
|
const sid = settings?.activeProfileId?.trim() ?? "";
|
||||||
if (sid && profiles.some((p) => p.id === sid)) return sid;
|
if (sid && profiles.some((profile) => profile.id === sid)) return sid;
|
||||||
return profiles[0]?.id ?? "";
|
return profiles[0]?.id ?? "";
|
||||||
|
}, [selectedId, settings?.activeProfileId, profiles]);
|
||||||
|
|
||||||
|
useEffect(() => {
|
||||||
|
const sid = settings?.activeProfileId?.trim() ?? "";
|
||||||
|
if (sid && profiles.some((profile) => profile.id === sid)) {
|
||||||
|
setSelectedId(sid);
|
||||||
|
}
|
||||||
}, [settings?.activeProfileId, profiles]);
|
}, [settings?.activeProfileId, profiles]);
|
||||||
|
|
||||||
const activateMutation = useMutation({
|
const activateMutation = useMutation({
|
||||||
mutationFn: (id: string) => api.activateProfile(id),
|
mutationFn: (id: string) => api.activateProfile(id),
|
||||||
onSuccess: async () => {
|
onSuccess: async (_data, profileId) => {
|
||||||
|
setSelectedId(profileId);
|
||||||
await Promise.all([
|
await Promise.all([
|
||||||
queryClient.invalidateQueries({ queryKey: queryKeys.jobs.all }),
|
queryClient.invalidateQueries({ queryKey: queryKeys.jobs.all }),
|
||||||
queryClient.invalidateQueries({ queryKey: queryKeys.profile.all }),
|
queryClient.invalidateQueries({ queryKey: queryKeys.profile.all }),
|
||||||
@@ -46,7 +67,10 @@ export const ProfileQuickSwitch: React.FC = () => {
|
|||||||
}),
|
}),
|
||||||
]);
|
]);
|
||||||
await refreshSettings();
|
await refreshSettings();
|
||||||
toast.success("Search profile updated");
|
const profileName =
|
||||||
|
profiles.find((profile) => profile.id === profileId)?.name ??
|
||||||
|
"Search profile";
|
||||||
|
toast.success(`${profileName} active`);
|
||||||
},
|
},
|
||||||
onError: (err: unknown) => {
|
onError: (err: unknown) => {
|
||||||
const message =
|
const message =
|
||||||
|
|||||||
@@ -199,6 +199,7 @@ export function buildMaxCoverageDiscoveryLimits(args: {
|
|||||||
builtinMaxJobsPerTerm: boardCap,
|
builtinMaxJobsPerTerm: boardCap,
|
||||||
builtinMaxPagesPerTerm: 10,
|
builtinMaxPagesPerTerm: 10,
|
||||||
wellfoundMaxJobsPerTerm: Math.min(100, boardCap),
|
wellfoundMaxJobsPerTerm: Math.min(100, boardCap),
|
||||||
|
googleJobsMaxJobsPerTerm: Math.min(30, boardCap),
|
||||||
qajobsboardMaxJobsPerTerm: boardCap,
|
qajobsboardMaxJobsPerTerm: boardCap,
|
||||||
arcMaxJobsPerPath: boardCap,
|
arcMaxJobsPerPath: boardCap,
|
||||||
smartrecruitersMaxJobsPerCompany: boardCap,
|
smartrecruitersMaxJobsPerCompany: boardCap,
|
||||||
|
|||||||
@@ -435,6 +435,11 @@ export const useOrchestratorFilters = () => {
|
|||||||
[searchParams],
|
[searchParams],
|
||||||
);
|
);
|
||||||
|
|
||||||
|
const hidePriorSkips = useMemo(
|
||||||
|
() => searchParams.get("hidePriorSkips") !== "0",
|
||||||
|
[searchParams],
|
||||||
|
);
|
||||||
|
|
||||||
const setApplySettingsCompanySkipList = useCallback(
|
const setApplySettingsCompanySkipList = useCallback(
|
||||||
(value: boolean) => {
|
(value: boolean) => {
|
||||||
setSearchParams(
|
setSearchParams(
|
||||||
@@ -449,6 +454,20 @@ export const useOrchestratorFilters = () => {
|
|||||||
[setSearchParams],
|
[setSearchParams],
|
||||||
);
|
);
|
||||||
|
|
||||||
|
const setHidePriorSkips = useCallback(
|
||||||
|
(value: boolean) => {
|
||||||
|
setSearchParams(
|
||||||
|
(prev) => {
|
||||||
|
if (value) prev.delete("hidePriorSkips");
|
||||||
|
else prev.set("hidePriorSkips", "0");
|
||||||
|
return prev;
|
||||||
|
},
|
||||||
|
{ replace: true },
|
||||||
|
);
|
||||||
|
},
|
||||||
|
[setSearchParams],
|
||||||
|
);
|
||||||
|
|
||||||
const sort = useMemo((): JobSort => {
|
const sort = useMemo((): JobSort => {
|
||||||
const sortValue = searchParams.get("sort");
|
const sortValue = searchParams.get("sort");
|
||||||
if (!sortValue) return DEFAULT_SORT;
|
if (!sortValue) return DEFAULT_SORT;
|
||||||
@@ -508,6 +527,7 @@ export const useOrchestratorFilters = () => {
|
|||||||
prev.delete("employer");
|
prev.delete("employer");
|
||||||
prev.delete("employerExclude");
|
prev.delete("employerExclude");
|
||||||
prev.delete("skipList");
|
prev.delete("skipList");
|
||||||
|
prev.delete("hidePriorSkips");
|
||||||
return prev;
|
return prev;
|
||||||
},
|
},
|
||||||
{ replace: true },
|
{ replace: true },
|
||||||
@@ -540,6 +560,8 @@ export const useOrchestratorFilters = () => {
|
|||||||
setEmployerFilterTokens,
|
setEmployerFilterTokens,
|
||||||
applySettingsCompanySkipList,
|
applySettingsCompanySkipList,
|
||||||
setApplySettingsCompanySkipList,
|
setApplySettingsCompanySkipList,
|
||||||
|
hidePriorSkips,
|
||||||
|
setHidePriorSkips,
|
||||||
sort,
|
sort,
|
||||||
setSort,
|
setSort,
|
||||||
resetFilters,
|
resetFilters,
|
||||||
|
|||||||
@@ -34,6 +34,7 @@ describe("orchestrator utils", () => {
|
|||||||
"testdevjobs",
|
"testdevjobs",
|
||||||
"wellfound",
|
"wellfound",
|
||||||
"builtin",
|
"builtin",
|
||||||
|
"google-jobs",
|
||||||
] as const) {
|
] as const) {
|
||||||
expect(enabled).toContain(source);
|
expect(enabled).toContain(source);
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -334,6 +334,7 @@ export const getEnabledSources = (
|
|||||||
source === "testdevjobs" ||
|
source === "testdevjobs" ||
|
||||||
source === "wellfound" ||
|
source === "wellfound" ||
|
||||||
source === "builtin" ||
|
source === "builtin" ||
|
||||||
|
source === "google-jobs" ||
|
||||||
source === "qajobsboard" ||
|
source === "qajobsboard" ||
|
||||||
source === "arcdev"
|
source === "arcdev"
|
||||||
) {
|
) {
|
||||||
|
|||||||
@@ -104,8 +104,8 @@ export const JobSourcesSettingsSection: React.FC<
|
|||||||
<p className="text-sm text-muted-foreground">
|
<p className="text-sm text-muted-foreground">
|
||||||
Configure company slugs, tenant IDs, and career-page URLs so ATS and
|
Configure company slugs, tenant IDs, and career-page URLs so ATS and
|
||||||
direct employer sources appear in the pipeline run dialog. Public job
|
direct employer sources appear in the pipeline run dialog. Public job
|
||||||
boards (Working Nomads, TestDevJobs, Wellfound, etc.) need no setup
|
boards (Working Nomads, TestDevJobs, Wellfound, Google Jobs, etc.)
|
||||||
here.
|
need no setup here.
|
||||||
</p>
|
</p>
|
||||||
|
|
||||||
<div className="space-y-4">
|
<div className="space-y-4">
|
||||||
|
|||||||
@@ -175,6 +175,10 @@ profilesRouter.post(
|
|||||||
const basicUser = parseBasicAuthUsername(req.headers.authorization)?.trim();
|
const basicUser = parseBasicAuthUsername(req.headers.authorization)?.trim();
|
||||||
|
|
||||||
if (isBasicAuthEnabled() && basicUser) {
|
if (isBasicAuthEnabled() && basicUser) {
|
||||||
|
await profilesRepo.setActiveSearchProfileIdForBasicAuthUser(
|
||||||
|
basicUser,
|
||||||
|
profile.id,
|
||||||
|
);
|
||||||
clearProfileCache();
|
clearProfileCache();
|
||||||
return ok(res, { activated: true, profileId: profile.id });
|
return ok(res, { activated: true, profileId: profile.id });
|
||||||
}
|
}
|
||||||
|
|||||||
@@ -271,6 +271,7 @@ export const DEMO_SOURCE_BASE_URLS: Record<JobSource, string> = {
|
|||||||
testdevjobs: "https://testdevjobs.com",
|
testdevjobs: "https://testdevjobs.com",
|
||||||
wellfound: "https://wellfound.com",
|
wellfound: "https://wellfound.com",
|
||||||
builtin: "https://builtin.com",
|
builtin: "https://builtin.com",
|
||||||
|
"google-jobs": "https://www.google.com/search?ibp=htl;jobs",
|
||||||
fourdayweek: "https://4dayweek.io",
|
fourdayweek: "https://4dayweek.io",
|
||||||
ashby: "https://jobs.ashbyhq.com",
|
ashby: "https://jobs.ashbyhq.com",
|
||||||
lever: "https://jobs.lever.co",
|
lever: "https://jobs.lever.co",
|
||||||
|
|||||||
@@ -29,6 +29,75 @@ function sqlInsertSearchProfileSeed(input: {
|
|||||||
return `INSERT OR IGNORE INTO search_profiles (id, name, data, created_at, updated_at) VALUES ('${id}', '${name}', '${data}', datetime('now'), datetime('now'))`;
|
return `INSERT OR IGNORE INTO search_profiles (id, name, data, created_at, updated_at) VALUES ('${id}', '${name}', '${data}', datetime('now'), datetime('now'))`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function sqlJsonLiteral(values: string[]): string {
|
||||||
|
return JSON.stringify(values).replace(/'/g, "''");
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Canada / remote — SDET & QA automation title variants for discovery + scoring. */
|
||||||
|
const ILIA_DOBKIN_CA_TARGET_ROLES = [
|
||||||
|
"SDET",
|
||||||
|
"Senior SDET",
|
||||||
|
"Lead SDET",
|
||||||
|
"Staff SDET",
|
||||||
|
"Principal SDET",
|
||||||
|
"Software Development Engineer in Test",
|
||||||
|
"Senior Software Development Engineer in Test",
|
||||||
|
"Software Engineer in Test",
|
||||||
|
"Software Test Engineer",
|
||||||
|
"Senior Software Test Engineer",
|
||||||
|
"QA Automation Engineer",
|
||||||
|
"Senior QA Automation Engineer",
|
||||||
|
"Quality Assurance Automation Engineer",
|
||||||
|
"Test Automation Engineer",
|
||||||
|
"Senior Test Automation Engineer",
|
||||||
|
"Automation Test Engineer",
|
||||||
|
"Automation Engineer",
|
||||||
|
"QA Engineer",
|
||||||
|
"Senior QA Engineer",
|
||||||
|
"Quality Engineer",
|
||||||
|
"Test Automation Lead",
|
||||||
|
"QA Automation Lead",
|
||||||
|
"Test Lead",
|
||||||
|
];
|
||||||
|
|
||||||
|
/** Job-board search terms — title variants plus common stack/tool pairings. */
|
||||||
|
const ILIA_DOBKIN_CA_KEYWORD_TERMS = [
|
||||||
|
...ILIA_DOBKIN_CA_TARGET_ROLES.map((role) => role.toLowerCase()),
|
||||||
|
"playwright SDET",
|
||||||
|
"SDET playwright",
|
||||||
|
"SDET java",
|
||||||
|
"SDET typescript",
|
||||||
|
"SDET cypress",
|
||||||
|
"SDET selenium",
|
||||||
|
"SDET api testing",
|
||||||
|
"SDET remote",
|
||||||
|
"QA automation remote",
|
||||||
|
"contract SDET",
|
||||||
|
"contract QA automation",
|
||||||
|
"SET",
|
||||||
|
"software engineer test",
|
||||||
|
];
|
||||||
|
|
||||||
|
const ILIA_DOBKIN_US_TARGET_ROLES = [
|
||||||
|
...ILIA_DOBKIN_CA_TARGET_ROLES,
|
||||||
|
"Computer Systems Analyst",
|
||||||
|
"Quality Assurance Engineer",
|
||||||
|
];
|
||||||
|
|
||||||
|
const ILIA_DOBKIN_US_KEYWORD_TERMS = [
|
||||||
|
...ILIA_DOBKIN_CA_KEYWORD_TERMS,
|
||||||
|
"computer systems analyst",
|
||||||
|
"systems analyst",
|
||||||
|
"computer systems analyst QA",
|
||||||
|
"SDET United States",
|
||||||
|
"SDET USA",
|
||||||
|
"SDET remote US",
|
||||||
|
"QA automation engineer USA",
|
||||||
|
"test automation engineer United States",
|
||||||
|
"quality assurance engineer automation",
|
||||||
|
"TN SDET",
|
||||||
|
];
|
||||||
|
|
||||||
const migrations = [
|
const migrations = [
|
||||||
`CREATE TABLE IF NOT EXISTS jobs (
|
`CREATE TABLE IF NOT EXISTS jobs (
|
||||||
id TEXT PRIMARY KEY,
|
id TEXT PRIMARY KEY,
|
||||||
@@ -817,28 +886,40 @@ const migrations = [
|
|||||||
id: "685b0000-0000-4000-8000-000000000001",
|
id: "685b0000-0000-4000-8000-000000000001",
|
||||||
name: "Ilia Dobkin",
|
name: "Ilia Dobkin",
|
||||||
profile: {
|
profile: {
|
||||||
targetRoles: [
|
targetRoles: ILIA_DOBKIN_CA_TARGET_ROLES,
|
||||||
"Software Development Engineer in Test",
|
|
||||||
"QA Automation Engineer",
|
|
||||||
"SDET",
|
|
||||||
],
|
|
||||||
experienceLevel: "Senior",
|
experienceLevel: "Senior",
|
||||||
mustHaveSkills: [
|
mustHaveSkills: [
|
||||||
"Playwright",
|
"Playwright",
|
||||||
"Cypress",
|
|
||||||
"Selenium",
|
|
||||||
"TypeScript",
|
"TypeScript",
|
||||||
|
"Java",
|
||||||
"API testing",
|
"API testing",
|
||||||
|
"CI/CD",
|
||||||
|
"Selenium",
|
||||||
|
"Cypress",
|
||||||
|
],
|
||||||
|
niceToHaveSkills: [
|
||||||
|
"Spring Boot",
|
||||||
|
"Python",
|
||||||
|
".NET",
|
||||||
|
"Contract testing",
|
||||||
|
"PostgreSQL",
|
||||||
|
"JUnit",
|
||||||
],
|
],
|
||||||
niceToHaveSkills: ["Python", ".NET", "CI/CD"],
|
|
||||||
dealBreakers: [],
|
dealBreakers: [],
|
||||||
preferredWorkArrangement: ["remote", "hybrid"],
|
preferredWorkArrangement: ["remote", "hybrid"],
|
||||||
preferredLocations: ["Toronto", "Ontario", "Canada"],
|
preferredLocations: [
|
||||||
|
"Toronto",
|
||||||
|
"Ontario",
|
||||||
|
"GTA",
|
||||||
|
"Greater Toronto Area",
|
||||||
|
"Canada",
|
||||||
|
"Remote Canada",
|
||||||
|
],
|
||||||
minimumSalary: "",
|
minimumSalary: "",
|
||||||
industriesToTarget: ["Software", "FinTech", "iGaming"],
|
industriesToTarget: ["Software", "FinTech", "iGaming", "Enterprise SaaS"],
|
||||||
industriesToAvoid: [],
|
industriesToAvoid: [],
|
||||||
aboutMe:
|
aboutMe:
|
||||||
"Ilia Dobkin — SDET / test automation; paired login user `ilia` activates this profile and local resume ilia-dobkin.json (unset JOBOPS_LOCAL_RESUME_PATH so Settings path applies).",
|
"Ilia Dobkin — Senior SDET / test automation (Canada & remote). Paired login user `ilia` activates this profile and local resume ilia-dobkin.json (unset JOBOPS_LOCAL_RESUME_PATH so Settings path applies).",
|
||||||
basicAuthUser: "ilia,dobkin",
|
basicAuthUser: "ilia,dobkin",
|
||||||
resumeLocalPath: "../data/resumes/ilia-dobkin.json",
|
resumeLocalPath: "../data/resumes/ilia-dobkin.json",
|
||||||
},
|
},
|
||||||
@@ -867,7 +948,7 @@ const migrations = [
|
|||||||
'ilia-dobkin-keyword-set',
|
'ilia-dobkin-keyword-set',
|
||||||
'685b0000-0000-4000-8000-000000000001',
|
'685b0000-0000-4000-8000-000000000001',
|
||||||
'SDET',
|
'SDET',
|
||||||
'["SDET","QA automation engineer","automation test engineer","playwright","software development engineer in test","quality assurance automation engineer"]',
|
'${sqlJsonLiteral(ILIA_DOBKIN_CA_KEYWORD_TERMS)}',
|
||||||
1,
|
1,
|
||||||
datetime('now'),
|
datetime('now'),
|
||||||
datetime('now')
|
datetime('now')
|
||||||
@@ -976,6 +1057,91 @@ const migrations = [
|
|||||||
updated_at = datetime('now')
|
updated_at = datetime('now')
|
||||||
WHERE id = '685b0000-0000-4000-8000-000000000002'
|
WHERE id = '685b0000-0000-4000-8000-000000000002'
|
||||||
AND coalesce(json_extract(data, '$.resumeLocalPath'), '') = ''`,
|
AND coalesce(json_extract(data, '$.resumeLocalPath'), '') = ''`,
|
||||||
|
sqlInsertSearchProfileSeed({
|
||||||
|
id: "685b0000-0000-4000-8000-000000000003",
|
||||||
|
name: "Ilia Dobkin (US)",
|
||||||
|
profile: {
|
||||||
|
targetRoles: ILIA_DOBKIN_US_TARGET_ROLES,
|
||||||
|
experienceLevel: "Senior",
|
||||||
|
mustHaveSkills: [
|
||||||
|
"Playwright",
|
||||||
|
"TypeScript",
|
||||||
|
"API testing",
|
||||||
|
"CI/CD",
|
||||||
|
"Java",
|
||||||
|
"Integration testing",
|
||||||
|
"Selenium",
|
||||||
|
],
|
||||||
|
niceToHaveSkills: [
|
||||||
|
"Cypress",
|
||||||
|
"Spring Boot",
|
||||||
|
"Python",
|
||||||
|
".NET",
|
||||||
|
"Contract testing",
|
||||||
|
"PostgreSQL",
|
||||||
|
],
|
||||||
|
dealBreakers: [],
|
||||||
|
preferredWorkArrangement: ["remote", "hybrid"],
|
||||||
|
preferredLocations: ["United States", "U.S.", "USA", "Remote US", "TN"],
|
||||||
|
minimumSalary: "",
|
||||||
|
industriesToTarget: [
|
||||||
|
"Software",
|
||||||
|
"FinTech",
|
||||||
|
"Financial services",
|
||||||
|
"Enterprise SaaS",
|
||||||
|
],
|
||||||
|
industriesToAvoid: [],
|
||||||
|
aboutMe:
|
||||||
|
"Ilia Dobkin (US) — Senior SDET / QA automation; Canadian citizen, TN-eligible (USMCA), open to U.S. relocation. Computer Systems Analyst-aligned framing. Switch to this profile in the Jobs page picker (same login as Canada).",
|
||||||
|
basicAuthUser: "ilia,dobkin",
|
||||||
|
resumeLocalPath: "../data/resumes/ilia-dobkin-us.json",
|
||||||
|
},
|
||||||
|
}),
|
||||||
|
`UPDATE search_profiles
|
||||||
|
SET data = json_set(data, '$.targetRoles', '${sqlJsonLiteral(ILIA_DOBKIN_CA_TARGET_ROLES)}'),
|
||||||
|
updated_at = datetime('now')
|
||||||
|
WHERE id = '685b0000-0000-4000-8000-000000000001'`,
|
||||||
|
`UPDATE keyword_sets
|
||||||
|
SET terms = '${sqlJsonLiteral(ILIA_DOBKIN_CA_KEYWORD_TERMS)}',
|
||||||
|
updated_at = datetime('now')
|
||||||
|
WHERE id = 'ilia-dobkin-keyword-set'`,
|
||||||
|
`INSERT OR IGNORE INTO keyword_sets (id, owner_profile_id, name, terms, is_active, created_at, updated_at)
|
||||||
|
VALUES (
|
||||||
|
'ilia-dobkin-us-keyword-set',
|
||||||
|
'685b0000-0000-4000-8000-000000000003',
|
||||||
|
'SDET',
|
||||||
|
'${sqlJsonLiteral(ILIA_DOBKIN_US_KEYWORD_TERMS)}',
|
||||||
|
1,
|
||||||
|
datetime('now'),
|
||||||
|
datetime('now')
|
||||||
|
)`,
|
||||||
|
`UPDATE search_profiles
|
||||||
|
SET data = json_set(
|
||||||
|
json_set(data, '$.basicAuthUser', 'ilia,dobkin'),
|
||||||
|
'$.aboutMe',
|
||||||
|
'Ilia Dobkin (US) — Senior SDET / QA automation; Canadian citizen, TN-eligible (USMCA), open to U.S. relocation. Computer Systems Analyst-aligned framing. Switch to this profile in the Jobs page picker (same login as Canada).'
|
||||||
|
),
|
||||||
|
updated_at = datetime('now')
|
||||||
|
WHERE id = '685b0000-0000-4000-8000-000000000003'`,
|
||||||
|
`UPDATE search_profiles
|
||||||
|
SET data = json_set(data, '$.targetRoles', '${sqlJsonLiteral(ILIA_DOBKIN_US_TARGET_ROLES)}'),
|
||||||
|
updated_at = datetime('now')
|
||||||
|
WHERE id = '685b0000-0000-4000-8000-000000000003'`,
|
||||||
|
`UPDATE keyword_sets
|
||||||
|
SET terms = '${sqlJsonLiteral(ILIA_DOBKIN_US_KEYWORD_TERMS)}',
|
||||||
|
updated_at = datetime('now')
|
||||||
|
WHERE id = 'ilia-dobkin-us-keyword-set'`,
|
||||||
|
// Remove duplicate Ilia profiles left by repository tests (same names, test UUIDs).
|
||||||
|
`DELETE FROM keyword_sets
|
||||||
|
WHERE owner_profile_id IN (
|
||||||
|
'685b0000-0000-4000-8000-000000009901',
|
||||||
|
'685b0000-0000-4000-8000-000000009902'
|
||||||
|
)`,
|
||||||
|
`DELETE FROM search_profiles
|
||||||
|
WHERE id IN (
|
||||||
|
'685b0000-0000-4000-8000-000000009901',
|
||||||
|
'685b0000-0000-4000-8000-000000009902'
|
||||||
|
)`,
|
||||||
];
|
];
|
||||||
|
|
||||||
console.log("🔧 Running database migrations...");
|
console.log("🔧 Running database migrations...");
|
||||||
|
|||||||
@@ -50,6 +50,9 @@ async function loadJobDedupIndexes(ownerProfileId: string): Promise<{
|
|||||||
existingContentFingerprintSet: Set<string>;
|
existingContentFingerprintSet: Set<string>;
|
||||||
dismissedDedupKeySet: Set<string>;
|
dismissedDedupKeySet: Set<string>;
|
||||||
}> {
|
}> {
|
||||||
|
const dismissLookbackMs = 90 * 24 * 60 * 60 * 1000;
|
||||||
|
const dismissCutoffMs = Date.now() - dismissLookbackMs;
|
||||||
|
|
||||||
const rows = await db
|
const rows = await db
|
||||||
.select({
|
.select({
|
||||||
jobUrl: jobs.jobUrl,
|
jobUrl: jobs.jobUrl,
|
||||||
@@ -60,6 +63,7 @@ async function loadJobDedupIndexes(ownerProfileId: string): Promise<{
|
|||||||
title: jobs.title,
|
title: jobs.title,
|
||||||
jobDescription: jobs.jobDescription,
|
jobDescription: jobs.jobDescription,
|
||||||
status: jobs.status,
|
status: jobs.status,
|
||||||
|
updatedAt: jobs.updatedAt,
|
||||||
})
|
})
|
||||||
.from(jobs)
|
.from(jobs)
|
||||||
.where(eq(jobs.ownerProfileId, ownerProfileId));
|
.where(eq(jobs.ownerProfileId, ownerProfileId));
|
||||||
@@ -94,6 +98,8 @@ async function loadJobDedupIndexes(ownerProfileId: string): Promise<{
|
|||||||
}
|
}
|
||||||
|
|
||||||
if (row.status === "skipped" || row.status === "applied") {
|
if (row.status === "skipped" || row.status === "applied") {
|
||||||
|
const updatedMs = Date.parse(row.updatedAt);
|
||||||
|
if (Number.isFinite(updatedMs) && updatedMs >= dismissCutoffMs) {
|
||||||
for (const key of collectJobDedupKeys({
|
for (const key of collectJobDedupKeys({
|
||||||
employer: row.employer,
|
employer: row.employer,
|
||||||
title: row.title,
|
title: row.title,
|
||||||
@@ -103,6 +109,7 @@ async function loadJobDedupIndexes(ownerProfileId: string): Promise<{
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
}
|
||||||
return {
|
return {
|
||||||
existingCanonicalSet,
|
existingCanonicalSet,
|
||||||
existingSourceJobKeySet,
|
existingSourceJobKeySet,
|
||||||
|
|||||||
@@ -0,0 +1,108 @@
|
|||||||
|
import { eq, inArray } from "drizzle-orm";
|
||||||
|
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||||
|
import { db } from "../db";
|
||||||
|
import { searchProfiles } from "../db/schema";
|
||||||
|
import {
|
||||||
|
getSearchProfileIdForBasicAuthUser,
|
||||||
|
listProfilesForBasicAuthUser,
|
||||||
|
profileMatchesBasicAuthUser,
|
||||||
|
setActiveSearchProfileIdForBasicAuthUser,
|
||||||
|
} from "./profiles";
|
||||||
|
|
||||||
|
const CA_PROFILE_ID = "685b0000-0000-4000-8000-000000009901";
|
||||||
|
const US_PROFILE_ID = "685b0000-0000-4000-8000-000000009902";
|
||||||
|
|
||||||
|
const sharedProfileData = {
|
||||||
|
targetRoles: ["SDET"],
|
||||||
|
experienceLevel: "Senior",
|
||||||
|
mustHaveSkills: [],
|
||||||
|
niceToHaveSkills: [],
|
||||||
|
dealBreakers: [],
|
||||||
|
preferredWorkArrangement: [],
|
||||||
|
preferredLocations: [],
|
||||||
|
minimumSalary: "",
|
||||||
|
industriesToTarget: [],
|
||||||
|
industriesToAvoid: [],
|
||||||
|
aboutMe: "",
|
||||||
|
basicAuthUser: "__test_profiles_repo__",
|
||||||
|
};
|
||||||
|
|
||||||
|
describe("profiles repository", () => {
|
||||||
|
afterEach(async () => {
|
||||||
|
await db
|
||||||
|
.delete(searchProfiles)
|
||||||
|
.where(inArray(searchProfiles.id, [CA_PROFILE_ID, US_PROFILE_ID]));
|
||||||
|
});
|
||||||
|
|
||||||
|
beforeEach(async () => {
|
||||||
|
await db
|
||||||
|
.delete(searchProfiles)
|
||||||
|
.where(inArray(searchProfiles.id, [CA_PROFILE_ID, US_PROFILE_ID]));
|
||||||
|
await db.insert(searchProfiles).values([
|
||||||
|
{
|
||||||
|
id: CA_PROFILE_ID,
|
||||||
|
name: "Test Profiles Repo CA",
|
||||||
|
data: JSON.stringify({
|
||||||
|
...sharedProfileData,
|
||||||
|
preferredLocations: ["Canada"],
|
||||||
|
resumeLocalPath: "../data/resumes/ilia-dobkin.json",
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
{
|
||||||
|
id: US_PROFILE_ID,
|
||||||
|
name: "Test Profiles Repo US",
|
||||||
|
data: JSON.stringify({
|
||||||
|
...sharedProfileData,
|
||||||
|
preferredLocations: ["United States"],
|
||||||
|
resumeLocalPath: "../data/resumes/ilia-dobkin-us.json",
|
||||||
|
}),
|
||||||
|
},
|
||||||
|
]);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("matches comma-separated basic auth aliases", () => {
|
||||||
|
expect(profileMatchesBasicAuthUser("ilia,dobkin", "ILIA")).toBe(true);
|
||||||
|
expect(profileMatchesBasicAuthUser("ilia,dobkin", "dobkin")).toBe(true);
|
||||||
|
expect(profileMatchesBasicAuthUser("ilia,dobkin", "cherepaha")).toBe(false);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("lists every profile visible to the login", async () => {
|
||||||
|
const visible = await listProfilesForBasicAuthUser(
|
||||||
|
"__test_profiles_repo__",
|
||||||
|
);
|
||||||
|
expect(visible.map((profile) => profile.id).sort()).toEqual(
|
||||||
|
[CA_PROFILE_ID, US_PROFILE_ID].sort(),
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("defaults to the first matching profile when none is stored", async () => {
|
||||||
|
await expect(
|
||||||
|
getSearchProfileIdForBasicAuthUser("__test_profiles_repo__"),
|
||||||
|
).resolves.toBe(CA_PROFILE_ID);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("remembers the active profile across basic auth aliases", async () => {
|
||||||
|
await db
|
||||||
|
.update(searchProfiles)
|
||||||
|
.set({
|
||||||
|
data: JSON.stringify({
|
||||||
|
...sharedProfileData,
|
||||||
|
preferredLocations: ["Canada"],
|
||||||
|
basicAuthUser: "test-repo-a,test-repo-b",
|
||||||
|
}),
|
||||||
|
})
|
||||||
|
.where(eq(searchProfiles.id, US_PROFILE_ID));
|
||||||
|
|
||||||
|
await setActiveSearchProfileIdForBasicAuthUser(
|
||||||
|
"test-repo-a",
|
||||||
|
US_PROFILE_ID,
|
||||||
|
);
|
||||||
|
|
||||||
|
await expect(
|
||||||
|
getSearchProfileIdForBasicAuthUser("test-repo-a"),
|
||||||
|
).resolves.toBe(US_PROFILE_ID);
|
||||||
|
await expect(
|
||||||
|
getSearchProfileIdForBasicAuthUser("test-repo-b"),
|
||||||
|
).resolves.toBe(US_PROFILE_ID);
|
||||||
|
});
|
||||||
|
});
|
||||||
@@ -8,6 +8,13 @@ import type {
|
|||||||
import { eq } from "drizzle-orm";
|
import { eq } from "drizzle-orm";
|
||||||
import { db } from "../db";
|
import { db } from "../db";
|
||||||
import { searchProfiles } from "../db/schema";
|
import { searchProfiles } from "../db/schema";
|
||||||
|
import type { SettingKey } from "./settings";
|
||||||
|
import { getSetting, setSetting } from "./settings";
|
||||||
|
|
||||||
|
const BASIC_AUTH_ACTIVE_PROFILE_BY_USER_KEY: SettingKey =
|
||||||
|
"basicAuthActiveProfileByUser";
|
||||||
|
|
||||||
|
type BasicAuthActiveProfileByUser = Record<string, string>;
|
||||||
|
|
||||||
function mapRow(row: typeof searchProfiles.$inferSelect): SearchProfile {
|
function mapRow(row: typeof searchProfiles.$inferSelect): SearchProfile {
|
||||||
let data: JobSearchProfile;
|
let data: JobSearchProfile;
|
||||||
@@ -110,16 +117,76 @@ export async function updateProfile(
|
|||||||
return getProfileById(id);
|
return getProfileById(id);
|
||||||
}
|
}
|
||||||
|
|
||||||
|
function parseBasicAuthActiveProfileByUser(
|
||||||
|
raw: string | null | undefined,
|
||||||
|
): BasicAuthActiveProfileByUser {
|
||||||
|
if (!raw?.trim()) return {};
|
||||||
|
try {
|
||||||
|
const parsed: unknown = JSON.parse(raw);
|
||||||
|
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
|
||||||
|
return {};
|
||||||
|
}
|
||||||
|
const out: BasicAuthActiveProfileByUser = {};
|
||||||
|
for (const [key, value] of Object.entries(parsed)) {
|
||||||
|
const alias = key.trim().toLowerCase();
|
||||||
|
const profileId = typeof value === "string" ? value.trim() : "";
|
||||||
|
if (alias && profileId) out[alias] = profileId;
|
||||||
|
}
|
||||||
|
return out;
|
||||||
|
} catch {
|
||||||
|
return {};
|
||||||
|
}
|
||||||
|
}
|
||||||
|
|
||||||
|
async function loadBasicAuthActiveProfileByUser(): Promise<BasicAuthActiveProfileByUser> {
|
||||||
|
const raw = await getSetting(BASIC_AUTH_ACTIVE_PROFILE_BY_USER_KEY);
|
||||||
|
return parseBasicAuthActiveProfileByUser(raw);
|
||||||
|
}
|
||||||
|
|
||||||
|
function basicAuthAliases(basicAuthUser: string | null | undefined): string[] {
|
||||||
|
return (basicAuthUser ?? "")
|
||||||
|
.split(",")
|
||||||
|
.map((part) => part.trim().toLowerCase())
|
||||||
|
.filter(Boolean);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* When one login maps to multiple search profiles, remember which profile is active.
|
||||||
|
* Writes the same profile id for every alias on that profile (e.g. ilia + dobkin).
|
||||||
|
*/
|
||||||
|
export async function setActiveSearchProfileIdForBasicAuthUser(
|
||||||
|
username: string,
|
||||||
|
profileId: string,
|
||||||
|
): Promise<void> {
|
||||||
|
const profile = await getProfileById(profileId);
|
||||||
|
if (!profile) return;
|
||||||
|
|
||||||
|
const requestAlias = username.trim().toLowerCase();
|
||||||
|
if (!requestAlias) return;
|
||||||
|
if (!profileMatchesBasicAuthUser(profile.data.basicAuthUser, requestAlias)) {
|
||||||
|
return;
|
||||||
|
}
|
||||||
|
|
||||||
|
const map = await loadBasicAuthActiveProfileByUser();
|
||||||
|
for (const alias of basicAuthAliases(profile.data.basicAuthUser)) {
|
||||||
|
map[alias] = profileId;
|
||||||
|
}
|
||||||
|
await setSetting(BASIC_AUTH_ACTIVE_PROFILE_BY_USER_KEY, JSON.stringify(map));
|
||||||
|
}
|
||||||
|
|
||||||
export async function getSearchProfileIdForBasicAuthUser(
|
export async function getSearchProfileIdForBasicAuthUser(
|
||||||
username: string,
|
username: string,
|
||||||
): Promise<string | null> {
|
): Promise<string | null> {
|
||||||
const profiles = await listProfiles();
|
const matches = await listProfilesForBasicAuthUser(username);
|
||||||
for (const p of profiles) {
|
if (matches.length === 0) return null;
|
||||||
if (profileMatchesBasicAuthUser(p.data.basicAuthUser, username)) {
|
if (matches.length === 1) return matches[0].id;
|
||||||
return p.id;
|
|
||||||
|
const map = await loadBasicAuthActiveProfileByUser();
|
||||||
|
const preferred = map[username.trim().toLowerCase()];
|
||||||
|
if (preferred && matches.some((profile) => profile.id === preferred)) {
|
||||||
|
return preferred;
|
||||||
}
|
}
|
||||||
}
|
return matches[0].id;
|
||||||
return null;
|
|
||||||
}
|
}
|
||||||
|
|
||||||
export async function deleteProfile(id: string): Promise<boolean> {
|
export async function deleteProfile(id: string): Promise<boolean> {
|
||||||
|
|||||||
@@ -151,19 +151,7 @@ export async function getEffectiveSettings(): Promise<AppSettings> {
|
|||||||
...envSettings,
|
...envSettings,
|
||||||
};
|
};
|
||||||
|
|
||||||
// In Basic Auth mode, the "active" search profile is derived from the request
|
|
||||||
// context (basic auth username → profile.basicAuthUser), not from the stored
|
|
||||||
// activeProfileId setting. Expose that derived id so the UI updates the right
|
|
||||||
// profile record on save.
|
|
||||||
const requestOwnerProfileId = getJobOwnerProfileId();
|
const requestOwnerProfileId = getJobOwnerProfileId();
|
||||||
if (
|
|
||||||
tenantJobSearchProfile &&
|
|
||||||
requestOwnerProfileId &&
|
|
||||||
requestOwnerProfileId !== DEFAULT_JOB_OWNER_PROFILE_ID &&
|
|
||||||
requestOwnerProfileId !== "__unmapped__"
|
|
||||||
) {
|
|
||||||
result.activeProfileId = requestOwnerProfileId;
|
|
||||||
}
|
|
||||||
|
|
||||||
const rawModel = overrides.model;
|
const rawModel = overrides.model;
|
||||||
const modelDef = settingsRegistry.model;
|
const modelDef = settingsRegistry.model;
|
||||||
@@ -254,6 +242,19 @@ export async function getEffectiveSettings(): Promise<AppSettings> {
|
|||||||
}
|
}
|
||||||
}
|
}
|
||||||
|
|
||||||
|
// In Basic Auth mode, the active search profile is derived from the request
|
||||||
|
// context (username → basicAuthActiveProfileByUser), not the stored
|
||||||
|
// activeProfileId row. Apply after the string-settings loop, which would
|
||||||
|
// otherwise overwrite this field from the global settings table.
|
||||||
|
if (
|
||||||
|
tenantJobSearchProfile &&
|
||||||
|
requestOwnerProfileId &&
|
||||||
|
requestOwnerProfileId !== DEFAULT_JOB_OWNER_PROFILE_ID &&
|
||||||
|
requestOwnerProfileId !== "__unmapped__"
|
||||||
|
) {
|
||||||
|
result.activeProfileId = requestOwnerProfileId;
|
||||||
|
}
|
||||||
|
|
||||||
// Always expose the effective base resume id for the active RxResume mode.
|
// Always expose the effective base resume id for the active RxResume mode.
|
||||||
result.rxresumeBaseResumeId = rxresumeBaseResumeId;
|
result.rxresumeBaseResumeId = rxresumeBaseResumeId;
|
||||||
|
|
||||||
|
|||||||
Generated
+35
@@ -336,6 +336,37 @@
|
|||||||
"undici-types": "~7.16.0"
|
"undici-types": "~7.16.0"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"extractors/google-jobs": {
|
||||||
|
"name": "google-jobs-extractor",
|
||||||
|
"version": "0.0.1",
|
||||||
|
"dependencies": {
|
||||||
|
"camoufox-js": "^0.8.0",
|
||||||
|
"job-ops-shared": "^1.0.0",
|
||||||
|
"playwright": "^1.57.0",
|
||||||
|
"tsx": "^4.4.0"
|
||||||
|
},
|
||||||
|
"devDependencies": {
|
||||||
|
"@types/node": "^24.0.0",
|
||||||
|
"typescript": "~5.9.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"extractors/google-jobs/node_modules/@types/node": {
|
||||||
|
"version": "24.13.3",
|
||||||
|
"resolved": "https://registry.npmjs.org/@types/node/-/node-24.13.3.tgz",
|
||||||
|
"integrity": "sha512-Dh8vAsV36ig5wa9OX4pXvMc9D3Veibfw2wix0CUwYODLD8nkj9UsLjASr49nPg+2eKzxhBV+v7L8pXvT4e639Q==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT",
|
||||||
|
"dependencies": {
|
||||||
|
"undici-types": "~7.18.0"
|
||||||
|
}
|
||||||
|
},
|
||||||
|
"extractors/google-jobs/node_modules/undici-types": {
|
||||||
|
"version": "7.18.2",
|
||||||
|
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.18.2.tgz",
|
||||||
|
"integrity": "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w==",
|
||||||
|
"dev": true,
|
||||||
|
"license": "MIT"
|
||||||
|
},
|
||||||
"extractors/gradcracker": {
|
"extractors/gradcracker": {
|
||||||
"name": "gradcracker-extractor",
|
"name": "gradcracker-extractor",
|
||||||
"version": "0.0.1",
|
"version": "0.0.1",
|
||||||
@@ -14253,6 +14284,10 @@
|
|||||||
"url": "https://github.com/sponsors/sindresorhus"
|
"url": "https://github.com/sponsors/sindresorhus"
|
||||||
}
|
}
|
||||||
},
|
},
|
||||||
|
"node_modules/google-jobs-extractor": {
|
||||||
|
"resolved": "extractors/google-jobs",
|
||||||
|
"link": true
|
||||||
|
},
|
||||||
"node_modules/gopd": {
|
"node_modules/gopd": {
|
||||||
"version": "1.2.0",
|
"version": "1.2.0",
|
||||||
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
|
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
|
||||||
|
|||||||
@@ -0,0 +1,30 @@
|
|||||||
|
#
|
||||||
|
# Jobber cron env — U.S. pipeline (optional separate cron host)
|
||||||
|
# Copy to /root/.jobber-cron-dobkin-us.env (chmod 600)
|
||||||
|
#
|
||||||
|
# Note: the U.S. profile is also available in the UI under the same ilia/dobkin
|
||||||
|
# login — use the Jobs page profile picker to switch. This env file is only
|
||||||
|
# needed if you want a dedicated cron that always runs the U.S. profile without
|
||||||
|
# switching in the UI (set BASIC_AUTH_ACTIVE_PROFILE_BY_USER or activate once).
|
||||||
|
#
|
||||||
|
# Used by: scripts/jobber-pipeline-telegram.sh
|
||||||
|
#
|
||||||
|
|
||||||
|
# JobOps base URL (where the app is reachable from the cron host)
|
||||||
|
JOBOPS_URL=http://127.0.0.1:3005
|
||||||
|
|
||||||
|
# Optional: limit number of jobs linked in Telegram message
|
||||||
|
JOB_TELEGRAM_MAX_JOBS=25
|
||||||
|
|
||||||
|
# Optional: comma-separated sources to run (leave empty to use server defaults)
|
||||||
|
# JOBBER_PIPELINE_SOURCES=usajobs,adzuna,remoteok,remotive
|
||||||
|
|
||||||
|
# Same login as Canada; activate the U.S. profile once in Settings/Jobs picker,
|
||||||
|
# or set basicAuthActiveProfileByUser in the DB for this user.
|
||||||
|
BASIC_AUTH_USER=dobkin
|
||||||
|
BASIC_AUTH_PASSWORD=CHANGEME
|
||||||
|
|
||||||
|
# Telegram bot + chat destination
|
||||||
|
TELEGRAM_BOT_TOKEN=CHANGEME
|
||||||
|
TELEGRAM_CHAT_ID=CHANGEME
|
||||||
|
|
||||||
@@ -340,6 +340,14 @@ const ALL_TARGETS: Target[] = [
|
|||||||
? undefined
|
? undefined
|
||||||
: "Camoufox not installed (run: npx camoufox-js fetch)",
|
: "Camoufox not installed (run: npx camoufox-js fetch)",
|
||||||
},
|
},
|
||||||
|
{
|
||||||
|
id: "google-jobs",
|
||||||
|
importPath: "../extractors/google-jobs/manifest",
|
||||||
|
settings: { googleJobsMaxJobsPerTerm: "5" },
|
||||||
|
skipReason: camoufoxInstalled()
|
||||||
|
? undefined
|
||||||
|
: "Camoufox not installed (run: npx camoufox-js fetch)",
|
||||||
|
},
|
||||||
{
|
{
|
||||||
id: "workday",
|
id: "workday",
|
||||||
importPath: "../extractors/workday/manifest",
|
importPath: "../extractors/workday/manifest",
|
||||||
|
|||||||
@@ -26,6 +26,7 @@ export const EXTRACTOR_SOURCE_IDS = [
|
|||||||
"testdevjobs",
|
"testdevjobs",
|
||||||
"wellfound",
|
"wellfound",
|
||||||
"builtin",
|
"builtin",
|
||||||
|
"google-jobs",
|
||||||
// --- Public ATS / career-page sources ---
|
// --- Public ATS / career-page sources ---
|
||||||
"ashby",
|
"ashby",
|
||||||
"lever",
|
"lever",
|
||||||
@@ -217,6 +218,12 @@ export const EXTRACTOR_SOURCE_METADATA: Record<
|
|||||||
category: "pipeline",
|
category: "pipeline",
|
||||||
region: "global",
|
region: "global",
|
||||||
},
|
},
|
||||||
|
"google-jobs": {
|
||||||
|
label: "Google Jobs",
|
||||||
|
order: 199,
|
||||||
|
category: "pipeline",
|
||||||
|
region: "global",
|
||||||
|
},
|
||||||
ashby: {
|
ashby: {
|
||||||
label: "Ashby (ATS)",
|
label: "Ashby (ATS)",
|
||||||
order: 210,
|
order: 210,
|
||||||
|
|||||||
@@ -3,8 +3,11 @@ import {
|
|||||||
buildJobContentFingerprint,
|
buildJobContentFingerprint,
|
||||||
buildJobDescriptionFingerprint,
|
buildJobDescriptionFingerprint,
|
||||||
collectJobDedupKeys,
|
collectJobDedupKeys,
|
||||||
|
employersMatchForDismiss,
|
||||||
|
jobsMatchEmployerAndTitleDismiss,
|
||||||
normalizeEmployerForFingerprint,
|
normalizeEmployerForFingerprint,
|
||||||
normalizeTitleForFingerprint,
|
normalizeTitleForFingerprint,
|
||||||
|
titlesMatchForDismiss,
|
||||||
} from "./job-fingerprint";
|
} from "./job-fingerprint";
|
||||||
|
|
||||||
describe("buildJobContentFingerprint", () => {
|
describe("buildJobContentFingerprint", () => {
|
||||||
@@ -96,10 +99,58 @@ describe("buildJobContentFingerprint", () => {
|
|||||||
expect(normalizeEmployerForFingerprint("Acme GmbH")).toBe("acme");
|
expect(normalizeEmployerForFingerprint("Acme GmbH")).toBe("acme");
|
||||||
});
|
});
|
||||||
|
|
||||||
|
it("normalizeEmployerForFingerprint strips leading punctuation", () => {
|
||||||
|
expect(normalizeEmployerForFingerprint(". CGI IT UK Limited")).toBe(
|
||||||
|
"cgiituklimited",
|
||||||
|
);
|
||||||
|
});
|
||||||
|
|
||||||
it("normalizeTitleForFingerprint drops leading repost markers", () => {
|
it("normalizeTitleForFingerprint drops leading repost markers", () => {
|
||||||
expect(normalizeTitleForFingerprint("[Reposted] Software Engineer")).toBe(
|
expect(normalizeTitleForFingerprint("[Reposted] Software Engineer")).toBe(
|
||||||
"softwareengineer",
|
"softwareengineer",
|
||||||
);
|
);
|
||||||
});
|
});
|
||||||
});
|
});
|
||||||
|
|
||||||
|
describe("prior skip dismiss matching", () => {
|
||||||
|
it("employersMatchForDismiss allows prefix matches", () => {
|
||||||
|
expect(
|
||||||
|
employersMatchForDismiss("cgi", normalizeEmployerForFingerprint("CGI")),
|
||||||
|
).toBe(true);
|
||||||
|
expect(
|
||||||
|
employersMatchForDismiss(
|
||||||
|
normalizeEmployerForFingerprint("CGI IT UK Limited"),
|
||||||
|
normalizeEmployerForFingerprint("CGI"),
|
||||||
|
),
|
||||||
|
).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("titlesMatchForDismiss allows plural variants", () => {
|
||||||
|
expect(
|
||||||
|
titlesMatchForDismiss(
|
||||||
|
normalizeTitleForFingerprint("Automation Test Engineer"),
|
||||||
|
normalizeTitleForFingerprint("Automation Test Engineers"),
|
||||||
|
),
|
||||||
|
).toBe(true);
|
||||||
|
});
|
||||||
|
|
||||||
|
it("jobsMatchEmployerAndTitleDismiss requires same title family", () => {
|
||||||
|
expect(
|
||||||
|
jobsMatchEmployerAndTitleDismiss({
|
||||||
|
employerA: "CGI IT UK Limited",
|
||||||
|
titleA: "Automation Test Engineers",
|
||||||
|
employerB: "CGI",
|
||||||
|
titleB: "Automation Test Engineer",
|
||||||
|
}),
|
||||||
|
).toBe(true);
|
||||||
|
expect(
|
||||||
|
jobsMatchEmployerAndTitleDismiss({
|
||||||
|
employerA: "Cognizant",
|
||||||
|
titleA: "AI Automation Test Engineer",
|
||||||
|
employerB: "Cognizant",
|
||||||
|
titleB: "Automation Test Engineer",
|
||||||
|
}),
|
||||||
|
).toBe(false);
|
||||||
|
});
|
||||||
|
});
|
||||||
});
|
});
|
||||||
|
|||||||
@@ -29,6 +29,9 @@ const TRAILING_CITY_REGION_RE = /\s+[-|–—]\s+[^,\-|–—]+,\s*[^,\-|–—]
|
|||||||
const COMPANY_LEGAL_SUFFIX_RE =
|
const COMPANY_LEGAL_SUFFIX_RE =
|
||||||
/\b(?:inc|inc\.|ltd|ltd\.|llc|gmbh|s\.a\.|s\.r\.l|sa|nv|bv|plc|corp|corporation|co|company|holdings|holding)\b/g;
|
/\b(?:inc|inc\.|ltd|ltd\.|llc|gmbh|s\.a\.|s\.r\.l|sa|nv|bv|plc|corp|corporation|co|company|holdings|holding)\b/g;
|
||||||
|
|
||||||
|
/** Hide rediscovered roles that match a prior skip/apply within this window. */
|
||||||
|
export const PRIOR_SKIP_DISMISS_LOOKBACK_MS = 90 * 24 * 60 * 60 * 1000;
|
||||||
|
|
||||||
function stripDiacritics(input: string): string {
|
function stripDiacritics(input: string): string {
|
||||||
return input.normalize("NFKD").replace(/[\u0300-\u036f]/g, "");
|
return input.normalize("NFKD").replace(/[\u0300-\u036f]/g, "");
|
||||||
}
|
}
|
||||||
@@ -38,6 +41,7 @@ export function normalizeEmployerForFingerprint(
|
|||||||
): string {
|
): string {
|
||||||
if (!employer) return "";
|
if (!employer) return "";
|
||||||
let value = stripDiacritics(employer.toLowerCase()).trim();
|
let value = stripDiacritics(employer.toLowerCase()).trim();
|
||||||
|
value = value.replace(/^[^a-z0-9]+/g, "");
|
||||||
value = value.replace(PARENS_RE, " ");
|
value = value.replace(PARENS_RE, " ");
|
||||||
value = value.replace(COMPANY_LEGAL_SUFFIX_RE, " ");
|
value = value.replace(COMPANY_LEGAL_SUFFIX_RE, " ");
|
||||||
value = value.replace(PUNCTUATION_RE, " ");
|
value = value.replace(PUNCTUATION_RE, " ");
|
||||||
@@ -103,6 +107,69 @@ export function buildJobDescriptionFingerprint(args: {
|
|||||||
return `${employer}::desc::${description}`;
|
return `${employer}::desc::${description}`;
|
||||||
}
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether two normalized employers refer to the same company for dismiss matching.
|
||||||
|
* Allows prefix matches so `cgi` matches `cgiituklimited`.
|
||||||
|
*/
|
||||||
|
export function employersMatchForDismiss(
|
||||||
|
normalizedEmployerA: string,
|
||||||
|
normalizedEmployerB: string,
|
||||||
|
): boolean {
|
||||||
|
if (!normalizedEmployerA || !normalizedEmployerB) return false;
|
||||||
|
if (normalizedEmployerA === normalizedEmployerB) return true;
|
||||||
|
const [short, long] =
|
||||||
|
normalizedEmployerA.length <= normalizedEmployerB.length
|
||||||
|
? [normalizedEmployerA, normalizedEmployerB]
|
||||||
|
: [normalizedEmployerB, normalizedEmployerA];
|
||||||
|
if (short.length < 3) return false;
|
||||||
|
return long.startsWith(short);
|
||||||
|
}
|
||||||
|
|
||||||
|
/**
|
||||||
|
* Whether two normalized titles are the same role for dismiss matching.
|
||||||
|
* Allows a trailing plural `s` variant (`engineer` vs `engineers`).
|
||||||
|
*/
|
||||||
|
export function titlesMatchForDismiss(
|
||||||
|
normalizedTitleA: string,
|
||||||
|
normalizedTitleB: string,
|
||||||
|
): boolean {
|
||||||
|
if (!normalizedTitleA || !normalizedTitleB) return false;
|
||||||
|
if (normalizedTitleA === normalizedTitleB) return true;
|
||||||
|
if (
|
||||||
|
normalizedTitleA.length > 4 &&
|
||||||
|
normalizedTitleA.endsWith("s") &&
|
||||||
|
normalizedTitleA.slice(0, -1) === normalizedTitleB
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
if (
|
||||||
|
normalizedTitleB.length > 4 &&
|
||||||
|
normalizedTitleB.endsWith("s") &&
|
||||||
|
normalizedTitleB.slice(0, -1) === normalizedTitleA
|
||||||
|
) {
|
||||||
|
return true;
|
||||||
|
}
|
||||||
|
return false;
|
||||||
|
}
|
||||||
|
|
||||||
|
/** Same company + same job title (normalized), for prior-skip hide rules. */
|
||||||
|
export function jobsMatchEmployerAndTitleDismiss(args: {
|
||||||
|
employerA: string | null | undefined;
|
||||||
|
titleA: string | null | undefined;
|
||||||
|
employerB: string | null | undefined;
|
||||||
|
titleB: string | null | undefined;
|
||||||
|
}): boolean {
|
||||||
|
const employerA = normalizeEmployerForFingerprint(args.employerA);
|
||||||
|
const employerB = normalizeEmployerForFingerprint(args.employerB);
|
||||||
|
const titleA = normalizeTitleForFingerprint(args.titleA);
|
||||||
|
const titleB = normalizeTitleForFingerprint(args.titleB);
|
||||||
|
if (!employerA || !employerB || !titleA || !titleB) return false;
|
||||||
|
return (
|
||||||
|
employersMatchForDismiss(employerA, employerB) &&
|
||||||
|
titlesMatchForDismiss(titleA, titleB)
|
||||||
|
);
|
||||||
|
}
|
||||||
|
|
||||||
export function collectJobDedupKeys(args: {
|
export function collectJobDedupKeys(args: {
|
||||||
employer: string | null | undefined;
|
employer: string | null | undefined;
|
||||||
title: string | null | undefined;
|
title: string | null | undefined;
|
||||||
|
|||||||
@@ -480,6 +480,13 @@ export const settingsRegistry = {
|
|||||||
parse: parseIntOrNull,
|
parse: parseIntOrNull,
|
||||||
serialize: serializeNullableNumber,
|
serialize: serializeNullableNumber,
|
||||||
},
|
},
|
||||||
|
googleJobsMaxJobsPerTerm: {
|
||||||
|
kind: "typed" as const,
|
||||||
|
schema: z.number().int().min(1).max(150),
|
||||||
|
default: (): number => 30,
|
||||||
|
parse: parseIntOrNull,
|
||||||
|
serialize: serializeNullableNumber,
|
||||||
|
},
|
||||||
fourdayweekMaxJobsPerTerm: {
|
fourdayweekMaxJobsPerTerm: {
|
||||||
kind: "typed" as const,
|
kind: "typed" as const,
|
||||||
schema: z.number().int().min(1).max(1000),
|
schema: z.number().int().min(1).max(1000),
|
||||||
@@ -965,6 +972,10 @@ export const settingsRegistry = {
|
|||||||
kind: "string" as const,
|
kind: "string" as const,
|
||||||
schema: z.string().trim().max(200),
|
schema: z.string().trim().max(200),
|
||||||
},
|
},
|
||||||
|
basicAuthActiveProfileByUser: {
|
||||||
|
kind: "string" as const,
|
||||||
|
schema: z.string().trim().max(10000),
|
||||||
|
},
|
||||||
rxresumeBaseResumeId: {
|
rxresumeBaseResumeId: {
|
||||||
kind: "string" as const,
|
kind: "string" as const,
|
||||||
schema: z.string().trim().max(200),
|
schema: z.string().trim().max(200),
|
||||||
|
|||||||
@@ -182,6 +182,7 @@ export const createAppSettings = (
|
|||||||
override: null,
|
override: null,
|
||||||
},
|
},
|
||||||
activeProfileId: null,
|
activeProfileId: null,
|
||||||
|
basicAuthActiveProfileByUser: null,
|
||||||
rxresumeBaseResumeId: null,
|
rxresumeBaseResumeId: null,
|
||||||
rxresumeBaseResumeIdV4: null,
|
rxresumeBaseResumeIdV4: null,
|
||||||
rxresumeBaseResumeIdV5: null,
|
rxresumeBaseResumeIdV5: null,
|
||||||
@@ -206,6 +207,7 @@ export const createAppSettings = (
|
|||||||
builtinMaxJobsPerTerm: { value: 100, default: 100, override: null },
|
builtinMaxJobsPerTerm: { value: 100, default: 100, override: null },
|
||||||
builtinMaxPagesPerTerm: { value: 3, default: 3, override: null },
|
builtinMaxPagesPerTerm: { value: 3, default: 3, override: null },
|
||||||
wellfoundMaxJobsPerTerm: { value: 50, default: 50, override: null },
|
wellfoundMaxJobsPerTerm: { value: 50, default: 50, override: null },
|
||||||
|
googleJobsMaxJobsPerTerm: { value: 30, default: 30, override: null },
|
||||||
fourdayweekMaxJobsPerTerm: { value: 50, default: 50, override: null },
|
fourdayweekMaxJobsPerTerm: { value: 50, default: 50, override: null },
|
||||||
qajobsboardMaxJobsPerTerm: { value: 100, default: 100, override: null },
|
qajobsboardMaxJobsPerTerm: { value: 100, default: 100, override: null },
|
||||||
arcRemoteJobsPaths: {
|
arcRemoteJobsPaths: {
|
||||||
|
|||||||
@@ -218,6 +218,7 @@ export interface AppSettings {
|
|||||||
builtinMaxJobsPerTerm: Resolved<number>;
|
builtinMaxJobsPerTerm: Resolved<number>;
|
||||||
builtinMaxPagesPerTerm: Resolved<number>;
|
builtinMaxPagesPerTerm: Resolved<number>;
|
||||||
wellfoundMaxJobsPerTerm: Resolved<number>;
|
wellfoundMaxJobsPerTerm: Resolved<number>;
|
||||||
|
googleJobsMaxJobsPerTerm: Resolved<number>;
|
||||||
fourdayweekMaxJobsPerTerm: Resolved<number>;
|
fourdayweekMaxJobsPerTerm: Resolved<number>;
|
||||||
qajobsboardMaxJobsPerTerm: Resolved<number>;
|
qajobsboardMaxJobsPerTerm: Resolved<number>;
|
||||||
arcRemoteJobsPaths: Resolved<string[]>;
|
arcRemoteJobsPaths: Resolved<string[]>;
|
||||||
@@ -270,6 +271,8 @@ export interface AppSettings {
|
|||||||
|
|
||||||
// Simple strings:
|
// Simple strings:
|
||||||
activeProfileId: string | null;
|
activeProfileId: string | null;
|
||||||
|
/** JSON map of basic-auth username → search profile id when one login has multiple profiles. */
|
||||||
|
basicAuthActiveProfileByUser: string | null;
|
||||||
rxresumeBaseResumeId: string | null;
|
rxresumeBaseResumeId: string | null;
|
||||||
rxresumeBaseResumeIdV4: string | null;
|
rxresumeBaseResumeIdV4: string | null;
|
||||||
rxresumeBaseResumeIdV5: string | null;
|
rxresumeBaseResumeIdV5: string | null;
|
||||||
|
|||||||
Reference in New Issue
Block a user