feat: add Google Jobs extractor and fix scraper reliability gaps
Ship a Camoufox-backed Google Jobs source, restore Glassdoor results discarded by a python-jobspy GraphQL quirk, and fix Himalayas/Gradcracker zero-job failures. Also harden prior-skip dismiss matching and multi-profile basic-auth switching for CA/US runs.
This commit is contained in:
parent
84e6835b11
commit
40c7cdece3
30
.env.example
30
.env.example
@ -260,7 +260,35 @@ ADZUNA_APP_KEY=
|
||||
# =============================================================================
|
||||
# Caps via Settings: workingnomadsMaxJobsPerTerm, testdevjobsMaxJobsPerTerm,
|
||||
# testdevjobsMaxPages, builtinMaxJobsPerTerm, builtinMaxPagesPerTerm,
|
||||
# wellfoundMaxJobsPerTerm (Wellfound uses Camoufox when installed).
|
||||
# wellfoundMaxJobsPerTerm, googleJobsMaxJobsPerTerm (Wellfound and Google
|
||||
# Jobs both use Camoufox when installed).
|
||||
|
||||
# =============================================================================
|
||||
# Google Jobs — optional (auto-enabled, no credentials, unofficial scrape)
|
||||
# =============================================================================
|
||||
# GOOGLE_JOBS_PROXY=http://user:pass@proxy-host:port
|
||||
# ^ Most reliable way to avoid Google's CAPTCHA/rate-limit wall (residential or
|
||||
# mobile proxy recommended; datacenter IPs, including most VPS/cloud hosts,
|
||||
# are aggressively blocked). Applies to both manual runs and the pipeline.
|
||||
#
|
||||
# GOOGLE_JOBS_HEADLESS=false
|
||||
# ^ Manual/local use ONLY (do not set this in the server .env for the
|
||||
# unattended pipeline/cron — a headless server has no display for a human
|
||||
# to see the browser). Opens a real Firefox window so you can solve a
|
||||
# CAPTCHA by hand; waits up to GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS (default
|
||||
# 300000 = 5 min) before giving up. Run directly, e.g.:
|
||||
# GOOGLE_JOBS_HEADLESS=false npm --workspace google-jobs-extractor run start
|
||||
# Session cookies are saved to extractors/google-jobs/storage/state.json
|
||||
# and reused by later runs (including headless/automated ones), which can
|
||||
# reduce how often future runs hit a CAPTCHA.
|
||||
#
|
||||
# Caps via Settings: googleJobsMaxJobsPerTerm (default 30, max 150).
|
||||
#
|
||||
# Note: there is no country/location override for this extractor — Google
|
||||
# resolves location from the browser's real (or GOOGLE_JOBS_PROXY's) IP.
|
||||
# Forcing a mismatched country has been observed to make Google's Jobs
|
||||
# vertical return "no matches" even though the tab loads correctly. To
|
||||
# target a different country's jobs, use a proxy with an exit node there.
|
||||
|
||||
# =============================================================================
|
||||
# Teamtailor / Huntflow / Factorial / Career pages — optional
|
||||
|
||||
@ -12,8 +12,9 @@
|
||||
"!!**/.venv",
|
||||
"!!docs-site/.docusaurus",
|
||||
"!!docs-site/build",
|
||||
"!!extractors/jobspy/storage",
|
||||
"!!extractors/*/storage",
|
||||
"!!orchestrator/storage",
|
||||
"!!storage",
|
||||
"!!data"
|
||||
]
|
||||
},
|
||||
|
||||
@ -32,10 +32,11 @@ Eluta surfaces Canadian roles indexed directly from employer career sites, often
|
||||
### Empty feeds
|
||||
|
||||
- The `location` string may be too broad or spelled differently than Eluta expects. Try a major city plus province (e.g. `Calgary, AB`).
|
||||
- Eluta's RSS is a small recent slice of *all* jobs for that location (often retail/hospitality-heavy). Pipeline search terms like `software engineer` can legitimately match zero items even when the feed itself is healthy. Broaden terms or add more metro locations.
|
||||
|
||||
### RSS HTTP errors
|
||||
### RSS HTTP errors / "Too Many Requests"
|
||||
|
||||
- Eluta may block unusual clients; the extractor sends a conventional User-Agent. Retry later or reduce the number of location feeds per run.
|
||||
- Eluta rate-limits aggressive or browser-like clients. The extractor uses a dedicated `JobOps/... Eluta RSS consumer` User-Agent; retry later or reduce the number of location feeds per run.
|
||||
|
||||
## Related pages
|
||||
|
||||
|
||||
106
docs-site/docs/extractors/google-jobs.md
Normal file
106
docs-site/docs/extractors/google-jobs.md
Normal file
@ -0,0 +1,106 @@
|
||||
---
|
||||
id: google-jobs
|
||||
title: Google Jobs Extractor
|
||||
description: Browser-backed scraping of the Google for Jobs search widget.
|
||||
sidebar_position: 8
|
||||
---
|
||||
|
||||
## What it is
|
||||
|
||||
Original feature: [Google for Jobs](https://www.google.com/search?q=jobs&udm=8)
|
||||
|
||||
Google Jobs is a browser-backed extractor that opens Google's Jobs vertical (`udm=8`) in Firefox (Camoufox when available) for each pipeline search term, scrolls the results panel to load additional cards, then clicks into each card to resolve a real "Apply on <site>" link and description before mapping rows into the orchestrator `CreateJobInput` shape.
|
||||
|
||||
Implementation split:
|
||||
|
||||
1. `extractors/google-jobs/src/main.ts` builds the search URL, waits for the jobs list, scrolls to lazy-load more cards, extracts card metadata (title/employer/location/"via" source), clicks each card to read the detail pane, and writes dataset JSON.
|
||||
2. `extractors/google-jobs/src/run.ts` spawns the browser subprocess, applies a hard timeout, reads the dataset, and maps rows to `CreateJobInput`.
|
||||
3. `extractors/google-jobs/manifest.ts` wires pipeline `searchTerms` into the extractor.
|
||||
|
||||
### A note on the search URL and location
|
||||
|
||||
Google's Jobs vertical is reached with `udm=8` (its current internal vertical id). An older, commonly-documented deep link — `ibp=htl;jobs` — still works but gets silently redirected to `udm=8` anyway, and combining it with extra parameters (a `num=` result-count hint, or a `gl=` country override that doesn't match the browser's real/proxied IP) has been observed to make Google return "no matches" or "can't determine location" even though the tab itself loads correctly.
|
||||
|
||||
Because of that, this extractor does **not** send a `gl` (country) parameter — location is resolved by Google from the browser's real IP (or `GOOGLE_JOBS_PROXY`'s exit IP), which Camoufox's `geoip` option keeps internally consistent. To target a different country's jobs, use a residential proxy with an exit node in that country rather than trying to override the country via a parameter.
|
||||
|
||||
### Dealing with Google's CAPTCHA / bot detection
|
||||
|
||||
Google actively fingerprints and challenges automated browser traffic, especially from datacenter/VPS IPs (which most servers running JobOps' cron pipeline will have). The extractor has layered mitigations, from always-on to opt-in:
|
||||
|
||||
1. **Always on:** Camoufox launches with `humanize: true` and `geoip: true` (human-like cursor movement, IP-consistent locale/geolocation) to reduce — not eliminate — fingerprint-based detection.
|
||||
2. **Always on:** browser cookies/session are persisted to `extractors/google-jobs/storage/state.json` and reused on the next run. A session that has cleared a CAPTCHA once (including via option 4 below) tends to see fewer challenges for a while afterward.
|
||||
3. **Opt-in, most reliable:** set `GOOGLE_JOBS_PROXY` (e.g. a residential/mobile proxy URL) to avoid datacenter-IP blocks in the first place. This is the standard real-world fix; unproxied runs from cloud servers should expect to be blocked often.
|
||||
4. **Opt-in, manual/local only:** set `GOOGLE_JOBS_HEADLESS=false` and run the extractor directly (not via the pipeline) to open a real, visible Firefox window. If Google shows a CAPTCHA, solve it yourself in that window — the script polls and automatically resumes once it clears (up to `GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS`, default 5 minutes). The resulting session cookies are saved and reused by later automated/headless runs. **Do not** set `GOOGLE_JOBS_HEADLESS=false` in the server's always-on `.env` — a headless server has no display for a human to interact with, and the run will just wait until the timeout.
|
||||
|
||||
## Why it exists
|
||||
|
||||
Google for Jobs aggregates postings across the open web — including boards and employer career pages that JobOps does not otherwise scrape. Adding it as a proper source closes some of the gap between "jobs found by manually Googling a role" and "jobs discovered by the pipeline."
|
||||
|
||||
It is **not** a general web crawler and does not replace dedicated extractors: it only surfaces whatever Google's Jobs widget chooses to show for a query, subject to Google's own ranking, freshness, and regional availability.
|
||||
|
||||
## How to use it
|
||||
|
||||
1. Open **Run jobs** and choose **Automatic**.
|
||||
2. **Google Jobs** is enabled by default in **Sources** (toggle it off if you do not want it for this run).
|
||||
3. Set your existing automatic run knobs:
|
||||
- `searchTerms` become the `"<term> jobs"` query sent to Google per term.
|
||||
- `googleJobsMaxJobsPerTerm` (Settings) caps jobs scraped per run (default `30`, max `150`).
|
||||
- Location is derived from the browser's real/proxied IP, not from a country setting — see [the note above](#a-note-on-the-search-url-and-location).
|
||||
4. Start the run and watch progress in the pipeline progress card.
|
||||
|
||||
Local run example (headless, no proxy):
|
||||
|
||||
```bash
|
||||
GOOGLE_JOBS_SEARCH_TERMS='["automation engineer"]' \
|
||||
GOOGLE_JOBS_MAX_JOBS='10' \
|
||||
npm --workspace google-jobs-extractor run start
|
||||
```
|
||||
|
||||
Local run example (headed, solve the CAPTCHA yourself if one appears):
|
||||
|
||||
```bash
|
||||
GOOGLE_JOBS_HEADLESS=false \
|
||||
GOOGLE_JOBS_SEARCH_TERMS='["automation engineer"]' \
|
||||
npm --workspace google-jobs-extractor run start
|
||||
```
|
||||
|
||||
Defaults and constraints:
|
||||
|
||||
- No credentials required; this is a direct HTML/DOM scrape of Google's public search results, not an official API (Google does not offer a free Google Jobs API).
|
||||
- Google's markup and CSS class names change frequently and are obfuscated; the scraper favors known current card classes (e.g. `.EimVGf`) with multiple fallback selectors and a leaf-node text heuristic, but selectors can still rot and return zero results until updated.
|
||||
- Every card carries Google's own `data-share-url` permalink for that specific listing, which the extractor uses as `jobUrl` — so a job stays unique and reviewable even if the "click into the card for a detail pane" step below fails for it.
|
||||
- Because resolving a real application link requires clicking each card in a real browser, this extractor is slower per job than API-backed sources — keep `googleJobsMaxJobsPerTerm` modest for frequent runs.
|
||||
- If no external "Apply on <site>" link is found for a card, the extractor falls back to Google's own permalink for that listing so it's still reviewable, but not directly one-click-applyable.
|
||||
- Automated, frequent, or high-volume scraping of Google search results may be against Google's Terms of Service; treat this extractor as best-effort and keep run frequency/volume conservative.
|
||||
- See [Dealing with Google's CAPTCHA / bot detection](#dealing-with-googles-captcha--bot-detection) above for proxy and interactive-solve options.
|
||||
|
||||
## Common problems
|
||||
|
||||
### Google Jobs returns 0 jobs for a term
|
||||
|
||||
The extractor logs one of two distinct messages to help tell these apart:
|
||||
|
||||
- `Google's Jobs tab reported no matches for "<term>"` — Google correctly landed on the Jobs tab but genuinely has no results for that query/locale. Try a broader, more common phrasing (e.g. `"automation engineer"` instead of a very narrow title).
|
||||
- `Jobs list did not appear for "<term>"` — an unrecognized page state (consent wall, layout change, or rotted selectors). The extractor saves a screenshot and full HTML dump to `extractors/google-jobs/storage/debug-last.png` / `debug-last.html` for inspection; compare it against a fresh manual search to see what changed.
|
||||
|
||||
### Results look sparse or missing company/location
|
||||
|
||||
- The list-view heuristics rely on leaf DOM nodes near the job title; if Google restructures the card markup, company/location extraction can degrade even when titles still work. This is expected best-effort behavior for an unofficial scrape.
|
||||
|
||||
### Run is slow
|
||||
|
||||
- Each job requires a real click + detail-pane render, unlike single-request API extractors. Lower `googleJobsMaxJobsPerTerm` or reduce the number of search terms for faster runs.
|
||||
|
||||
### Blocked, CAPTCHA'd, or consistently empty results
|
||||
|
||||
- Google may rate-limit or challenge automated browser traffic, especially from datacenter/VPS IPs. In order of effort:
|
||||
1. Set `GOOGLE_JOBS_PROXY` to a residential/mobile proxy — the most reliable fix for unattended/server runs.
|
||||
2. Run locally with `GOOGLE_JOBS_HEADLESS=false` and solve the CAPTCHA yourself once; the saved session (`storage/state.json`) may reduce blocks on subsequent headless runs for a while.
|
||||
3. Reduce run frequency, lower the per-term cap, or disable the source for that run and rely on other extractors / Manual Import instead.
|
||||
|
||||
## Related pages
|
||||
|
||||
- [Extractors Overview](/docs/next/extractors/overview)
|
||||
- [Manual Import](/docs/next/extractors/manual)
|
||||
- [Pipeline Run](/docs/next/features/pipeline-run)
|
||||
- [Settings](/docs/next/features/settings)
|
||||
@ -46,3 +46,11 @@ Set `JOBOPS_EMIT_PROGRESS=1` for structured progress lines consumable by orchest
|
||||
|
||||
- Uses Playwright + Crawlee via Camoufox.
|
||||
- Low concurrency and longer timeouts for stability.
|
||||
- Employer logo (`figure img` alt text) and employer link are optional. Cards that omit them used to throw a Playwright `getAttribute` timeout and fail the whole region page (0 jobs saved even after cards were counted). Missing employer fields are now skipped per card.
|
||||
|
||||
## Common problems
|
||||
|
||||
### Region pages time out / run returns 0 jobs
|
||||
|
||||
- Gradcracker Livewire markup sometimes omits the employer logo on a card. Older builds aborted the whole page on that timeout. Current builds continue and leave employer blank.
|
||||
- Anti-bot / Camoufox issues: ensure Camoufox is installed (`npx camoufox-js fetch`) and prefer headed runs when debugging.
|
||||
|
||||
45
docs-site/docs/extractors/himalayas.md
Normal file
45
docs-site/docs/extractors/himalayas.md
Normal file
@ -0,0 +1,45 @@
|
||||
---
|
||||
id: himalayas
|
||||
title: Himalayas Extractor
|
||||
description: Public remote-jobs API pagination and client-side term filtering.
|
||||
sidebar_position: 16
|
||||
---
|
||||
|
||||
## What it is
|
||||
|
||||
Original site: [himalayas.app](https://himalayas.app)
|
||||
|
||||
The extractor lives in `extractors/himalayas/manifest.ts`. It paginates the public JSON API (`https://himalayas.app/jobs/api?limit=&offset=`), filters rows client-side by pipeline search terms (title + categories), and maps matches into `CreateJobInput`.
|
||||
|
||||
## Why it exists
|
||||
|
||||
Himalayas is a large remote-jobs index with a stable, unauthenticated API — useful for remote-first discovery without browser automation or credentials.
|
||||
|
||||
## How to use it
|
||||
|
||||
1. Enable **Himalayas** in pipeline sources.
|
||||
2. Optionally set **Himalayas max jobs per term** (`himalayasMaxJobsPerTerm`, default `100`).
|
||||
3. Pipeline `searchTerms` filter titles/categories client-side (substring match, case-insensitive).
|
||||
4. Run the pipeline.
|
||||
|
||||
### Defaults and constraints
|
||||
|
||||
- The upstream API **silently caps `limit` at 20** per request. The extractor uses `PAGE_SIZE = 20` and paginates with `offset` (up to 10 pages / 200 rows scanned per run before the per-term cap).
|
||||
- There is no server-side search — if a term is rare, early pages may not contain matches even though later pages do. Raise the page budget only by changing the extractor constants if you need deeper scans.
|
||||
- No auth required.
|
||||
|
||||
## Common problems
|
||||
|
||||
### Zero jobs for a term I know exists
|
||||
|
||||
- The term may not appear in the first ~200 API rows (newest-first feed). Try a shorter token (`software`, `engineer`, `SDET`) or accept that Himalayas skews toward whatever is currently at the top of the feed.
|
||||
- Confirm you are not hitting an old build that requested `limit=50` and stopped after one page (fixed: the API returns at most 20, which used to look like end-of-feed).
|
||||
|
||||
### Rate limits / HTTP errors
|
||||
|
||||
- Retry later; the public API is generally stable but can throttle burst traffic.
|
||||
|
||||
## Related pages
|
||||
|
||||
- [Extractors Overview](/docs/next/extractors/overview)
|
||||
- [Add an Extractor](/docs/next/workflows/add-an-extractor)
|
||||
@ -61,4 +61,8 @@ The service in `orchestrator/src/server/services/jobspy.ts`:
|
||||
- A run returned fewer LinkedIn descriptions than expected.
|
||||
`JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0` disables description fetching to speed up runs.
|
||||
- Different cities need different workplace-type filters.
|
||||
This is not supported in the current automatic-run flow. JobSpy receives one global workplace-type selection per run/query invocation.
|
||||
This is not supported in the current automatic-run flow. JobSpy receives one global workplace-type selection per run/query invocation.
|
||||
- Glassdoor always returns 0 jobs (fixed).
|
||||
Upstream `python-jobspy` (through at least `1.1.82`) discards an entire page of Glassdoor results whenever Glassdoor's GraphQL response includes *any* `errors` entry — even for an unrelated field. In practice, Glassdoor's backend consistently returns a partial error for `jobsPageSeoData` (`SUBREQUEST_HTTP_ERROR` / "dns error" on Glassdoor's own infra) while the actual job listings still come back fine, so every Glassdoor run silently returned 0 jobs. `extractors/jobspy/scrape_jobs.py` monkeypatches `Glassdoor._fetch_jobs_page` at import time to only treat the response as failed when `data.jobListings` is actually missing, and logs a warning (not an error) when it ignores an unrelated partial error. `requirements.txt` pins `python-jobspy==1.1.82` so an upstream version bump doesn't silently drop this patch — re-verify the patched method still matches before removing the pin.
|
||||
- A Glassdoor run logs `429` / "Blocked by Glassdoor for too many requests".
|
||||
This is a real rate limit from running too many Glassdoor requests back-to-back (e.g. repeated manual test runs). Space out runs; the automatic pipeline's normal cadence does not trigger this in practice.
|
||||
@ -25,6 +25,7 @@ Extractor integrations are now registered through manifests and loaded automatic
|
||||
| [Eluta](/docs/next/extractors/eluta) | Canadian listings aggregated from employer career sites (RSS) | Canada-only source (skipped when search geography is not Canada); RSS `location` strings must be set | `ELUTA_RSS_LOCATIONS`, `ELUTA_MAX_JOBS_PER_TERM` | Fetches one or more `eluta.ca` RSS feeds, filters by terms, de-duplicates by guid/URL |
|
||||
| [QAJobsBoard](/docs/next/extractors/qajobsboard) | QA / SDET / automation-heavy board (global JSON feed) | No auth; geography skew is manual/filter downstream | `qajobsboardMaxJobsPerTerm` | Fetches JobBoardly JSON, filters by pipeline terms |
|
||||
| [Arc.dev](/docs/next/extractors/arcdev) | Remote roles from Arc.dev listing pages (tool-tagged paths) | Parses SSR `__NEXT_DATA__`; relies on stable Next payload | `ARC_REMOTE_JOBS_PATHS` (seeds defaults), `arcRemoteJobsPaths`, `arcMaxJobsPerPath` | Merges Arc-managed + external rows; dedupes by URL |
|
||||
| [Google Jobs](/docs/next/extractors/google-jobs) | Broader open-web coverage via the Google for Jobs widget | Unofficial scrape (no API); CAPTCHA-prone from datacenter IPs; selectors can rot; slower (click-through per job) | `googleJobsMaxJobsPerTerm`, `GOOGLE_JOBS_PROXY`, `GOOGLE_JOBS_HEADLESS` | Browser scrape of `udm=8` Jobs vertical; scrolls list, double-clicks cards, expands description |
|
||||
| [Manual Import](/docs/next/extractors/manual) | One-off jobs not covered by scrapers | Inference quality depends on model/provider and input quality; some URLs cannot be fetched reliably | App/API endpoints (`/api/manual-jobs/infer`, `/api/manual-jobs/import`) | Accepts text/HTML/URL, runs inference, then saves and scores job after review |
|
||||
|
||||
## Which extractor should I use?
|
||||
@ -40,6 +41,7 @@ Extractor integrations are now registered through manifests and loaded automatic
|
||||
- Use **BC T-Net** for British Columbia tech RSS listings (runs only when search geography is Canada).
|
||||
- Use **Eluta** for Canadian employer-direct listings via RSS (set metro/province `location` strings).
|
||||
- Use **QAJobsBoard** or **Arc.dev** when you want QA- or remote-stack-focused feeds without extra credentials.
|
||||
- Use **Google Jobs** when you want broader open-web coverage beyond dedicated job boards (unofficial scrape; best-effort).
|
||||
- Use **Manual Import** when you already have a specific posting and need direct import.
|
||||
|
||||
Many runs combine sources: broad discovery first, then manual import for high-priority jobs that scraping misses.
|
||||
@ -82,5 +84,7 @@ JobOps ships **BC T-Net** and **iCIMS tenant HTML** extractors for two cases tha
|
||||
- [Arc.dev](/docs/next/extractors/arcdev)
|
||||
- [Canadian / NA QA contracting firms](/docs/next/extractors/qa-contract-staffing-canada)
|
||||
- [Canadian companies — QA-strong ATS](/docs/next/extractors/canadian-companies-qa-ats)
|
||||
- [Google Jobs](/docs/next/extractors/google-jobs)
|
||||
- [Himalayas](/docs/next/extractors/himalayas)
|
||||
- [Manual Import](/docs/next/extractors/manual)
|
||||
- [Add an Extractor](/docs/next/workflows/add-an-extractor)
|
||||
|
||||
@ -45,9 +45,19 @@ During a pipeline run, if a new posting matches an existing row by URL, source i
|
||||
|
||||
Open jobs that match a prior skip or apply are **hidden** from Discovered, Ready, and All tabs so the queue stays fresh. Skipped and applied rows themselves remain visible in their statuses.
|
||||
|
||||
Use **Filters → Employer keywords → Hide roles you already skipped** to toggle this behavior (on by default). When enabled, JobOps compares **company + job title** only:
|
||||
|
||||
- Same company after normalization (legal suffixes stripped; short names like `CGI` match longer forms like `CGI IT UK Limited`).
|
||||
- Same title after normalization (including simple plural variants such as `Engineer` vs `Engineers`).
|
||||
- Prior skip or apply must be within the **last 90 days** (`updatedAt` on the skipped/applied row).
|
||||
|
||||
Uncheck the filter to temporarily review rediscovered rows that match older skips.
|
||||
|
||||
## Defaults and constraints
|
||||
|
||||
- Description matching requires at least **80 characters** of normalized text; short or empty descriptions fall back to employer+title only.
|
||||
- Prior-skip hiding uses employer + title only (not description text).
|
||||
- The 90-day window applies to both the Jobs list filter and pipeline import suppression for skipped/applied rows.
|
||||
- Description matching during import still requires at least **80 characters** of normalized text; short or empty descriptions fall back to employer+title only.
|
||||
- Matching is **per profile** (`ownerProfileId`); different login profiles do not share dedup state.
|
||||
- Dedup does **not** delete existing rows retroactively when you change skip list or country filters — run discovery again or skip manually for old data.
|
||||
- Very different titles at the same company (for example `SDET` vs `Product Designer`) are **not** collapsed.
|
||||
|
||||
49
extractors/google-jobs/manifest.ts
Normal file
49
extractors/google-jobs/manifest.ts
Normal file
@ -0,0 +1,49 @@
|
||||
import type {
|
||||
ExtractorManifest,
|
||||
ExtractorRunResult,
|
||||
} from "@shared/types/extractors";
|
||||
import { runGoogleJobs } from "./src/run.js";
|
||||
|
||||
export const manifest: ExtractorManifest = {
|
||||
id: "google-jobs",
|
||||
displayName: "Google Jobs",
|
||||
providesSources: ["google-jobs"],
|
||||
async run(context): Promise<ExtractorRunResult> {
|
||||
if (context.shouldCancel?.()) return { success: true, jobs: [] };
|
||||
|
||||
const parsedMax = context.settings.googleJobsMaxJobsPerTerm
|
||||
? Number.parseInt(context.settings.googleJobsMaxJobsPerTerm, 10)
|
||||
: Number.NaN;
|
||||
const maxJobs = Number.isFinite(parsedMax) ? Math.max(1, parsedMax) : 30;
|
||||
|
||||
context.onProgress?.({
|
||||
phase: "list",
|
||||
termsProcessed: 0,
|
||||
termsTotal: context.searchTerms.length || 1,
|
||||
currentUrl: "https://www.google.com/search?udm=8",
|
||||
detail: "Google Jobs: launching browser scrape",
|
||||
});
|
||||
|
||||
const result = await runGoogleJobs({
|
||||
searchTerms: context.searchTerms,
|
||||
maxJobs,
|
||||
});
|
||||
|
||||
if (!result.success) {
|
||||
return { success: false, jobs: [], error: result.error };
|
||||
}
|
||||
|
||||
context.onProgress?.({
|
||||
phase: "list",
|
||||
termsProcessed: context.searchTerms.length || 1,
|
||||
termsTotal: context.searchTerms.length || 1,
|
||||
currentUrl: "https://www.google.com/search?udm=8",
|
||||
jobPagesProcessed: result.jobs.length,
|
||||
detail: `Google Jobs: ${result.jobs.length} jobs`,
|
||||
});
|
||||
|
||||
return { success: true, jobs: result.jobs };
|
||||
},
|
||||
};
|
||||
|
||||
export default manifest;
|
||||
22
extractors/google-jobs/package.json
Normal file
22
extractors/google-jobs/package.json
Normal file
@ -0,0 +1,22 @@
|
||||
{
|
||||
"name": "google-jobs-extractor",
|
||||
"version": "0.0.1",
|
||||
"type": "module",
|
||||
"description": "Google Jobs (Google for Jobs) search widget extractor (browser-backed)",
|
||||
"main": "manifest.ts",
|
||||
"dependencies": {
|
||||
"camoufox-js": "^0.8.0",
|
||||
"job-ops-shared": "^1.0.0",
|
||||
"playwright": "^1.57.0",
|
||||
"tsx": "^4.4.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^24.0.0",
|
||||
"typescript": "~5.9.0"
|
||||
},
|
||||
"scripts": {
|
||||
"start": "tsx src/main.ts",
|
||||
"check:types": "tsc --noEmit",
|
||||
"get-binaries": "camoufox-js fetch"
|
||||
}
|
||||
}
|
||||
616
extractors/google-jobs/src/main.ts
Normal file
616
extractors/google-jobs/src/main.ts
Normal file
@ -0,0 +1,616 @@
|
||||
import { access, mkdir, writeFile } from "node:fs/promises";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { launchOptions } from "camoufox-js";
|
||||
import {
|
||||
type Browser,
|
||||
type BrowserContext,
|
||||
firefox,
|
||||
type Page,
|
||||
} from "playwright";
|
||||
|
||||
const __dirname = dirname(fileURLToPath(import.meta.url));
|
||||
const OUTPUT_PATH = join(__dirname, "../storage/jobs.json");
|
||||
const STATE_PATH = join(__dirname, "../storage/state.json");
|
||||
const DEBUG_HTML_PATH = join(__dirname, "../storage/debug-last.html");
|
||||
const DEBUG_SCREENSHOT_PATH = join(__dirname, "../storage/debug-last.png");
|
||||
|
||||
interface ScrapedJob {
|
||||
title: string;
|
||||
employer: string;
|
||||
jobUrl: string;
|
||||
applicationLink?: string;
|
||||
location?: string;
|
||||
jobDescription?: string;
|
||||
postedVia?: string;
|
||||
isRemote?: boolean;
|
||||
}
|
||||
|
||||
interface ScrapedCard {
|
||||
docId?: string;
|
||||
title: string;
|
||||
employer: string;
|
||||
location?: string;
|
||||
via?: string;
|
||||
/** Google's own permalink into this specific listing (from `data-share-url`). Always present when the card matched, so it works as a stable jobUrl even if the detail click fails. */
|
||||
shareUrl?: string;
|
||||
}
|
||||
|
||||
interface ScrapedDetail {
|
||||
applyLinks: Array<{ href: string; text: string }>;
|
||||
description?: string;
|
||||
}
|
||||
|
||||
/**
|
||||
* Confirmed against Google's live markup (2026). Google renders each job as
|
||||
* a `.EimVGf` card with a `data-share-url` permalink baked in — no `<ul>`/
|
||||
* `<li>`/`aria-label="Jobs list"` wrapper as older scraping guides assumed.
|
||||
* Kept as a short list (not a single hardcoded class) since Google rotates
|
||||
* these obfuscated names periodically.
|
||||
*/
|
||||
const CARD_SELECTORS = [".EimVGf", 'li[role="listitem"]', ".iFjolb"] as const;
|
||||
|
||||
function parseTerms(raw: string | undefined): string[] {
|
||||
if (!raw) return ["software engineer"];
|
||||
try {
|
||||
const parsed = JSON.parse(raw) as unknown;
|
||||
if (Array.isArray(parsed)) {
|
||||
return parsed.map((entry) => String(entry).trim()).filter(Boolean);
|
||||
}
|
||||
} catch {
|
||||
// fall through to delimited parsing
|
||||
}
|
||||
return raw
|
||||
.split(/[\n|,]+/)
|
||||
.map((entry) => entry.trim())
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
/**
|
||||
* `udm=8` is Google's current internal id for the Jobs vertical (the legacy
|
||||
* `ibp=htl;jobs` deep link now gets silently redirected to it anyway, but
|
||||
* only when requested directly — combining the two, or adding `num=`/`gl=`
|
||||
* overrides that don't match the browser's real/proxied IP, has been
|
||||
* observed to make the vertical come back "no matches" or "can't determine
|
||||
* location" even though the widget itself loads correctly). `hl=en` keeps
|
||||
* copy in English so downstream text parsing (e.g. "via LinkedIn") is
|
||||
* consistent regardless of where the request is geolocated. Location is
|
||||
* intentionally NOT set via a `gl` param — it's derived from the real (or
|
||||
* `GOOGLE_JOBS_PROXY`) IP through Camoufox's `geoip` option instead.
|
||||
*/
|
||||
function buildSearchUrl(term: string): string {
|
||||
const params = new URLSearchParams({
|
||||
q: `${term} jobs`,
|
||||
udm: "8",
|
||||
hl: "en",
|
||||
});
|
||||
return `https://www.google.com/search?${params.toString()}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Launches Camoufox-hardened Firefox. `humanize`/`geoip` reduce (but do not
|
||||
* eliminate) Google's automated-traffic detection. `GOOGLE_JOBS_PROXY` (e.g.
|
||||
* a residential proxy URL) is the most reliable way to avoid datacenter-IP
|
||||
* blocks entirely, if configured.
|
||||
*/
|
||||
async function launchBrowser(headless: boolean): Promise<Browser> {
|
||||
const proxy = process.env.GOOGLE_JOBS_PROXY?.trim();
|
||||
return firefox.launch(
|
||||
await launchOptions({
|
||||
headless,
|
||||
humanize: true,
|
||||
geoip: true,
|
||||
...(proxy ? { proxy } : {}),
|
||||
}),
|
||||
);
|
||||
}
|
||||
|
||||
/** Google shows a cookie/consent wall in some locales before rendering results. */
|
||||
async function dismissConsent(page: Page): Promise<void> {
|
||||
const labels = ["Reject all", "I agree", "Accept all"];
|
||||
for (const label of labels) {
|
||||
try {
|
||||
const button = page.getByRole("button", { name: label }).first();
|
||||
if (await button.isVisible({ timeout: 1500 })) {
|
||||
await button.click({ timeout: 3000 });
|
||||
await page.waitForTimeout(500);
|
||||
return;
|
||||
}
|
||||
} catch {
|
||||
// selector not present in this render; try the next label
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
/** Thrown when Google serves a CAPTCHA / rate-limit challenge instead of results. */
|
||||
class GoogleBlockedError extends Error {}
|
||||
|
||||
/** Google serves an interstitial CAPTCHA page for automated / datacenter traffic. */
|
||||
async function isBlockedPage(page: Page): Promise<boolean> {
|
||||
if (/\/sorry\//.test(page.url())) return true;
|
||||
try {
|
||||
return await page.evaluate(() => {
|
||||
const bodyText = document.body?.textContent ?? "";
|
||||
return (
|
||||
document.querySelector("#captcha-form") !== null ||
|
||||
document.querySelector(".g-recaptcha") !== null ||
|
||||
/unusual traffic/i.test(bodyText)
|
||||
);
|
||||
});
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* In headed/interactive mode, give a human at the keyboard a chance to solve
|
||||
* the CAPTCHA in the visible browser window instead of failing immediately.
|
||||
* Polls until the challenge clears (Google's form redirects on success) or
|
||||
* the timeout elapses.
|
||||
*/
|
||||
async function waitForHumanToUnblock(
|
||||
page: Page,
|
||||
timeoutMs: number,
|
||||
): Promise<boolean> {
|
||||
const deadline = Date.now() + timeoutMs;
|
||||
console.error(
|
||||
`[google-jobs] Google is showing a CAPTCHA in the open browser window — please solve it there. Waiting up to ${Math.round(timeoutMs / 1000)}s...`,
|
||||
);
|
||||
while (Date.now() < deadline) {
|
||||
await page.waitForTimeout(3_000);
|
||||
if (!(await isBlockedPage(page))) {
|
||||
console.error("[google-jobs] CAPTCHA cleared, resuming.");
|
||||
return true;
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
async function waitForJobsList(page: Page): Promise<boolean> {
|
||||
for (const selector of CARD_SELECTORS) {
|
||||
try {
|
||||
await page.waitForSelector(selector, { timeout: 8_000 });
|
||||
return true;
|
||||
} catch {
|
||||
// try the next fallback selector
|
||||
}
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Google sometimes lands correctly on the Jobs vertical/tab but reports no
|
||||
* matches for the query (distinct from a CAPTCHA block or a selector-rot
|
||||
* failure) — e.g. "It looks like there aren't any 'Jobs' matches on this topic."
|
||||
*/
|
||||
async function isEmptyJobsVertical(page: Page): Promise<boolean> {
|
||||
try {
|
||||
return await page.evaluate(() => {
|
||||
const bodyText = document.body?.textContent ?? "";
|
||||
return /aren.t any .Jobs. matches/i.test(bodyText);
|
||||
});
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
function countCardsScript(selectors: readonly string[]): number {
|
||||
for (const selector of selectors) {
|
||||
const count = document.querySelectorAll(selector).length;
|
||||
if (count > 0) return count;
|
||||
}
|
||||
return 0;
|
||||
}
|
||||
|
||||
/** Google lazy-loads additional cards (via a custom `<infinity-scrolling>` element) as the page is scrolled. */
|
||||
function scrollPageScript(): number {
|
||||
const before = window.scrollY;
|
||||
window.scrollTo(0, document.body.scrollHeight);
|
||||
return window.scrollY - before;
|
||||
}
|
||||
|
||||
async function growJobsList(
|
||||
page: Page,
|
||||
selectors: readonly string[],
|
||||
targetCount: number,
|
||||
): Promise<void> {
|
||||
for (let attempt = 0; attempt < 8; attempt += 1) {
|
||||
const count = await page.evaluate(countCardsScript, selectors);
|
||||
if (count >= targetCount) return;
|
||||
const scrolled = await page.evaluate(scrollPageScript);
|
||||
if (scrolled <= 0) return;
|
||||
await page.waitForTimeout(900);
|
||||
}
|
||||
}
|
||||
|
||||
function extractCardsScript(selectors: readonly string[]): ScrapedCard[] {
|
||||
// NOTE: this whole function is serialized via Function.prototype.toString()
|
||||
// and re-run inside the browser page by Playwright — it cannot reference
|
||||
// anything from the outer module scope (constants, other functions, etc.).
|
||||
// All helpers must be declared inside this function body.
|
||||
function clean(value: string | null | undefined): string {
|
||||
return (value ?? "").replace(/\s+/g, " ").trim();
|
||||
}
|
||||
|
||||
/** Extracts Google's internal `htidocid` from a job's `data-share-url`, matching `data-encoded-docid` in the detail panel's apply-link list. */
|
||||
function extractDocId(shareUrl: string | null): string | undefined {
|
||||
if (!shareUrl) return undefined;
|
||||
try {
|
||||
const url = new URL(shareUrl, "https://www.google.com");
|
||||
return url.searchParams.get("htidocid") ?? undefined;
|
||||
} catch {
|
||||
return undefined;
|
||||
}
|
||||
}
|
||||
|
||||
function findCards(): Element[] {
|
||||
for (const selector of selectors) {
|
||||
const items = Array.from(document.querySelectorAll(selector));
|
||||
if (items.length > 0) return items;
|
||||
}
|
||||
return [];
|
||||
}
|
||||
|
||||
const cards: ScrapedCard[] = [];
|
||||
|
||||
for (const item of findCards()) {
|
||||
// Known current classes first (title/employer/location+via), falling
|
||||
// back to a leaf-node text heuristic if Google has rotated them.
|
||||
const titleEl = item.querySelector(".tNxQIb, h2, [role='heading']");
|
||||
const title = clean(titleEl?.textContent);
|
||||
if (!title) continue;
|
||||
|
||||
const wHYlTdEls = Array.from(item.querySelectorAll(".wHYlTd"));
|
||||
let employer = clean(wHYlTdEls[0]?.textContent);
|
||||
let location: string | undefined;
|
||||
let via: string | undefined;
|
||||
|
||||
const viaBlock = wHYlTdEls.find((el) =>
|
||||
/\bvia\b/i.test(el.textContent ?? ""),
|
||||
);
|
||||
if (viaBlock) {
|
||||
const text = clean(viaBlock.textContent);
|
||||
const match = text.match(/^(.*?)\s*(?:•|·)\s*via\s+(.+)$/i);
|
||||
if (match) {
|
||||
location = match[1] || undefined;
|
||||
via = match[2] || undefined;
|
||||
} else {
|
||||
via = clean(text.replace(/^via\s*/i, ""));
|
||||
}
|
||||
}
|
||||
|
||||
if (!employer || !title || employer === title) {
|
||||
// Selector rot fallback: scan leaf text nodes structurally.
|
||||
const leafTexts = Array.from(item.querySelectorAll("div, span"))
|
||||
.filter((el) => el.children.length === 0)
|
||||
.map((el) => clean(el.textContent))
|
||||
.filter((text) => text && text !== title && !/^[•·|,]+$/.test(text));
|
||||
employer = employer || leafTexts[0] || "Unknown Employer";
|
||||
if (!via) {
|
||||
const viaText = leafTexts.find((text) => /^via\b/i.test(text));
|
||||
via = viaText ? clean(viaText.replace(/^via\s*/i, "")) : undefined;
|
||||
location =
|
||||
location ?? leafTexts.slice(1).find((text) => text !== viaText);
|
||||
}
|
||||
}
|
||||
|
||||
const shareUrlRaw = item.getAttribute("data-share-url") ?? undefined;
|
||||
const docId = extractDocId(shareUrlRaw ?? null);
|
||||
|
||||
cards.push({
|
||||
docId,
|
||||
title,
|
||||
employer: employer || "Unknown Employer",
|
||||
location,
|
||||
via,
|
||||
shareUrl: shareUrlRaw,
|
||||
});
|
||||
}
|
||||
|
||||
return cards;
|
||||
}
|
||||
|
||||
/**
|
||||
* Scopes apply-link extraction to the clicked job's own `data-encoded-docid`
|
||||
* block. Google's detail viewer can have adjacent/related jobs' apply links
|
||||
* present in the DOM simultaneously, so a document-wide "first apply link"
|
||||
* scan can silently pick up the wrong job's link.
|
||||
*/
|
||||
function extractDetailScript(docId: string | undefined): ScrapedDetail {
|
||||
function clean(value: string | null | undefined): string {
|
||||
return (value ?? "").replace(/\s+/g, " ").trim();
|
||||
}
|
||||
|
||||
function collectApplyLinks(
|
||||
root: ParentNode,
|
||||
): Array<{ href: string; text: string }> {
|
||||
const links: Array<{ href: string; text: string }> = [];
|
||||
for (const anchor of Array.from(root.querySelectorAll("a[href]"))) {
|
||||
const href = (anchor as HTMLAnchorElement).href;
|
||||
const text =
|
||||
clean(anchor.textContent) || clean(anchor.getAttribute("aria-label"));
|
||||
if (!href) continue;
|
||||
if (
|
||||
!/apply/i.test(text) &&
|
||||
!/apply/i.test(anchor.getAttribute("aria-label") ?? "")
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
if (
|
||||
href.includes("google.com/search") ||
|
||||
href.includes("accounts.google.com")
|
||||
) {
|
||||
continue;
|
||||
}
|
||||
links.push({ href, text });
|
||||
}
|
||||
return links;
|
||||
}
|
||||
|
||||
let applyLinks: Array<{ href: string; text: string }> = [];
|
||||
if (docId) {
|
||||
const scoped = document.querySelector(`[data-encoded-docid="${docId}"]`);
|
||||
if (scoped) applyLinks = collectApplyLinks(scoped);
|
||||
}
|
||||
if (applyLinks.length === 0) {
|
||||
applyLinks = collectApplyLinks(document);
|
||||
}
|
||||
|
||||
// `.textContent` also pulls in text from nested <style>/<script> tags
|
||||
// (Google embeds scoped theme <style> blocks inside content divs) and from
|
||||
// the job-list sidebar itself (which concatenates every visible card's
|
||||
// text and easily outsizes a real description) — exclude both.
|
||||
const blocks = Array.from(document.querySelectorAll("div"))
|
||||
.filter(
|
||||
(el) =>
|
||||
el.children.length <= 3 && !el.querySelector("style, script, .EimVGf"),
|
||||
)
|
||||
.map((el) => clean(el.textContent))
|
||||
.filter((text) => text.length > 200);
|
||||
blocks.sort((a, b) => b.length - a.length);
|
||||
|
||||
return {
|
||||
applyLinks,
|
||||
description: blocks[0]?.slice(0, 5_000),
|
||||
};
|
||||
}
|
||||
|
||||
async function openCardAndExtractDetail(
|
||||
page: Page,
|
||||
selectors: readonly string[],
|
||||
index: number,
|
||||
docId: string | undefined,
|
||||
): Promise<ScrapedDetail | null> {
|
||||
const elementHandle = await page.evaluateHandle(
|
||||
(args: { selectors: readonly string[]; idx: number }) => {
|
||||
for (const selector of args.selectors) {
|
||||
const items = document.querySelectorAll(selector);
|
||||
if (items.length > 0) return items[args.idx] ?? null;
|
||||
}
|
||||
return null;
|
||||
},
|
||||
{ selectors, idx: index },
|
||||
);
|
||||
|
||||
const element = elementHandle.asElement();
|
||||
if (!element) return null;
|
||||
|
||||
await element.scrollIntoViewIfNeeded();
|
||||
// A single click is unreliable here: clicking a card while a *different*
|
||||
// card's detail pane is already open frequently just closes the open pane
|
||||
// instead of switching to the new one (confirmed by watching real runs —
|
||||
// it alternates open/closed rather than opening the newly clicked card).
|
||||
// A double-click reopens reliably; verified across 6 sequential cards.
|
||||
await element.dblclick({ timeout: 5_000, delay: 80 });
|
||||
if (docId) {
|
||||
try {
|
||||
await page.waitForSelector(`[data-encoded-docid="${docId}"]`, {
|
||||
timeout: 6_000,
|
||||
});
|
||||
} catch {
|
||||
// fall through — extractDetailScript falls back to a document-wide
|
||||
// scan for apply links if the docid-scoped block never shows up.
|
||||
}
|
||||
}
|
||||
await page.waitForTimeout(500);
|
||||
return page.evaluate(extractDetailScript, docId);
|
||||
}
|
||||
|
||||
async function saveDebugArtifacts(page: Page): Promise<void> {
|
||||
try {
|
||||
await mkdir(dirname(DEBUG_HTML_PATH), { recursive: true });
|
||||
await writeFile(DEBUG_HTML_PATH, await page.content(), "utf-8");
|
||||
await page.screenshot({ path: DEBUG_SCREENSHOT_PATH, fullPage: true });
|
||||
} catch (error) {
|
||||
console.error(
|
||||
`[google-jobs] failed to save debug artifacts: ${error instanceof Error ? error.message : String(error)}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
async function scrapeTerm(
|
||||
context: BrowserContext,
|
||||
term: string,
|
||||
remainingCap: number,
|
||||
options: { headless: boolean; unblockTimeoutMs: number },
|
||||
): Promise<ScrapedJob[]> {
|
||||
const page = await context.newPage();
|
||||
const out: ScrapedJob[] = [];
|
||||
|
||||
try {
|
||||
await page.goto(buildSearchUrl(term), {
|
||||
waitUntil: "domcontentloaded",
|
||||
timeout: 45_000,
|
||||
});
|
||||
await dismissConsent(page);
|
||||
await page.waitForTimeout(1_500);
|
||||
|
||||
if (await isBlockedPage(page)) {
|
||||
const solvedByHuman =
|
||||
!options.headless &&
|
||||
(await waitForHumanToUnblock(page, options.unblockTimeoutMs));
|
||||
if (!solvedByHuman) {
|
||||
throw new GoogleBlockedError(
|
||||
"Google blocked this request (CAPTCHA / unusual traffic detection). This is common from shared, VPN, or datacenter IPs. Try again later, reduce run frequency, run from a residential network, set GOOGLE_JOBS_PROXY, or set GOOGLE_JOBS_HEADLESS=false to solve it yourself.",
|
||||
);
|
||||
}
|
||||
}
|
||||
|
||||
const listReady = await waitForJobsList(page);
|
||||
if (!listReady) {
|
||||
if (await isEmptyJobsVertical(page)) {
|
||||
console.error(
|
||||
`[google-jobs] Google's Jobs tab reported no matches for "${term}" (not a block or selector issue — try a broader/more common phrasing).`,
|
||||
);
|
||||
} else {
|
||||
console.error(
|
||||
`[google-jobs] Jobs list did not appear for "${term}" (url: ${page.url()}). Saved a screenshot + HTML dump to ${DEBUG_SCREENSHOT_PATH} for inspection.`,
|
||||
);
|
||||
await saveDebugArtifacts(page);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
await growJobsList(page, CARD_SELECTORS, remainingCap);
|
||||
|
||||
const cards = await page.evaluate(extractCardsScript, CARD_SELECTORS);
|
||||
const limit = Math.min(cards.length, remainingCap);
|
||||
|
||||
for (let index = 0; index < limit; index += 1) {
|
||||
const card = cards[index];
|
||||
// Google's own permalink into this specific listing — always available
|
||||
// from the list view, so results stay unique/useful even if the detail
|
||||
// click below fails for this card.
|
||||
const shareUrl = card.shareUrl;
|
||||
try {
|
||||
const detail = await openCardAndExtractDetail(
|
||||
page,
|
||||
CARD_SELECTORS,
|
||||
index,
|
||||
card.docId,
|
||||
);
|
||||
const applyLink = detail?.applyLinks[0];
|
||||
const jobUrl = shareUrl ?? applyLink?.href ?? page.url();
|
||||
|
||||
out.push({
|
||||
title: card.title,
|
||||
employer: card.employer,
|
||||
jobUrl,
|
||||
applicationLink: applyLink?.href,
|
||||
location: card.location,
|
||||
jobDescription: detail?.description,
|
||||
postedVia: card.via,
|
||||
isRemote: /remote/i.test(`${card.location ?? ""} ${card.title}`),
|
||||
});
|
||||
} catch (error) {
|
||||
console.error(
|
||||
`[google-jobs] card ${index} for "${term}" failed: ${
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}`,
|
||||
);
|
||||
if (shareUrl) {
|
||||
// Detail click failed, but we still have a unique, valid link to
|
||||
// the listing from the card itself — don't drop the job entirely.
|
||||
out.push({
|
||||
title: card.title,
|
||||
employer: card.employer,
|
||||
jobUrl: shareUrl,
|
||||
location: card.location,
|
||||
postedVia: card.via,
|
||||
isRemote: /remote/i.test(`${card.location ?? ""} ${card.title}`),
|
||||
});
|
||||
}
|
||||
}
|
||||
await page.waitForTimeout(300 + Math.floor(Math.random() * 400));
|
||||
}
|
||||
} finally {
|
||||
await page.close();
|
||||
}
|
||||
|
||||
return out;
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
const terms = parseTerms(process.env.GOOGLE_JOBS_SEARCH_TERMS);
|
||||
const maxJobsEnv = Number.parseInt(
|
||||
process.env.GOOGLE_JOBS_MAX_JOBS ?? "30",
|
||||
10,
|
||||
);
|
||||
const cap = Number.isFinite(maxJobsEnv)
|
||||
? Math.max(1, Math.min(maxJobsEnv, 150))
|
||||
: 30;
|
||||
// Headed mode lets a human solve a CAPTCHA in the visible window; only
|
||||
// useful when run directly on a machine with a display, not on a headless
|
||||
// cron/server pipeline.
|
||||
const headless = process.env.GOOGLE_JOBS_HEADLESS !== "false";
|
||||
const unblockTimeoutMs = Number.parseInt(
|
||||
process.env.GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS ?? "300000",
|
||||
10,
|
||||
);
|
||||
|
||||
const browser = await launchBrowser(headless);
|
||||
const hasSavedState = await access(STATE_PATH)
|
||||
.then(() => true)
|
||||
.catch(() => false);
|
||||
const context = await browser.newContext(
|
||||
hasSavedState ? { storageState: STATE_PATH } : {},
|
||||
);
|
||||
// tsx/esbuild compiles this file with `keepNames`, which injects
|
||||
// `__name(fn, "fn")` calls into any named function it transpiles —
|
||||
// including ones later serialized into `page.evaluate()` scripts via
|
||||
// `Function.prototype.toString()`. That helper only exists in our Node
|
||||
// process, not in the browser page, so without this no-op polyfill any
|
||||
// evaluate script with a nested named function throws
|
||||
// `ReferenceError: __name is not defined`.
|
||||
await context.addInitScript(() => {
|
||||
(window as unknown as { __name?: (fn: unknown) => unknown }).__name = (
|
||||
fn: unknown,
|
||||
) => fn;
|
||||
});
|
||||
|
||||
const all: ScrapedJob[] = [];
|
||||
const seenUrls = new Set<string>();
|
||||
|
||||
try {
|
||||
for (const term of terms) {
|
||||
if (all.length >= cap) break;
|
||||
try {
|
||||
const rows = await scrapeTerm(context, term, cap - all.length, {
|
||||
headless,
|
||||
unblockTimeoutMs,
|
||||
});
|
||||
for (const row of rows) {
|
||||
if (all.length >= cap) break;
|
||||
if (seenUrls.has(row.jobUrl)) continue;
|
||||
seenUrls.add(row.jobUrl);
|
||||
all.push(row);
|
||||
}
|
||||
} catch (error) {
|
||||
if (error instanceof GoogleBlockedError) {
|
||||
// IP-level block affects the whole run; stop trying further terms
|
||||
// and let it propagate so the process exits non-zero with a clear message.
|
||||
throw error;
|
||||
}
|
||||
console.error(
|
||||
`[google-jobs] term "${term}" failed: ${
|
||||
error instanceof Error ? error.message : String(error)
|
||||
}`,
|
||||
);
|
||||
}
|
||||
}
|
||||
} finally {
|
||||
// Persist cookies/session (even a partially solved one) so future
|
||||
// headless/automated runs can reuse it and hit fewer CAPTCHA walls.
|
||||
await mkdir(dirname(STATE_PATH), { recursive: true });
|
||||
await context.storageState({ path: STATE_PATH }).catch(() => undefined);
|
||||
await browser.close();
|
||||
}
|
||||
|
||||
await mkdir(dirname(OUTPUT_PATH), { recursive: true });
|
||||
await writeFile(OUTPUT_PATH, JSON.stringify(all, null, 2), "utf-8");
|
||||
}
|
||||
|
||||
main().catch((error) => {
|
||||
console.error(error instanceof Error ? error.message : String(error));
|
||||
process.exit(1);
|
||||
});
|
||||
154
extractors/google-jobs/src/run.ts
Normal file
154
extractors/google-jobs/src/run.ts
Normal file
@ -0,0 +1,154 @@
|
||||
import { spawn, spawnSync } from "node:child_process";
|
||||
import { readFile } from "node:fs/promises";
|
||||
import { createRequire } from "node:module";
|
||||
import { dirname, join } from "node:path";
|
||||
import { fileURLToPath } from "node:url";
|
||||
import { envForExtractorSubprocess } from "@shared/extractor-subprocess-env.js";
|
||||
import type { CreateJobInput } from "@shared/types/jobs";
|
||||
|
||||
const srcDir = dirname(fileURLToPath(import.meta.url));
|
||||
const EXTRACTOR_DIR = join(srcDir, "..");
|
||||
const OUTPUT_PATH = join(EXTRACTOR_DIR, "storage/jobs.json");
|
||||
const require = createRequire(import.meta.url);
|
||||
const TSX_CLI_PATH = (() => {
|
||||
try {
|
||||
return require.resolve("tsx/dist/cli.mjs");
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
})();
|
||||
|
||||
/** Browser scrape can take a while (per-term navigation + per-card clicks). */
|
||||
const SUBPROCESS_TIMEOUT_MS = 4 * 60 * 1000;
|
||||
|
||||
interface ScrapedJob {
|
||||
title?: string;
|
||||
employer?: string;
|
||||
jobUrl?: string;
|
||||
applicationLink?: string;
|
||||
location?: string;
|
||||
jobDescription?: string;
|
||||
postedVia?: string;
|
||||
isRemote?: boolean;
|
||||
}
|
||||
|
||||
export interface RunGoogleJobsOptions {
|
||||
searchTerms?: string[];
|
||||
maxJobs?: number;
|
||||
}
|
||||
|
||||
export interface GoogleJobsResult {
|
||||
success: boolean;
|
||||
jobs: CreateJobInput[];
|
||||
error?: string;
|
||||
}
|
||||
|
||||
function canRunNpmCommand(): boolean {
|
||||
const result = spawnSync("npm", ["--version"], { stdio: "ignore" });
|
||||
return !result.error && result.status === 0;
|
||||
}
|
||||
|
||||
function mapJob(row: ScrapedJob): CreateJobInput | null {
|
||||
const jobUrl = row.jobUrl?.trim();
|
||||
if (!jobUrl) return null;
|
||||
return {
|
||||
source: "google-jobs",
|
||||
title: row.title?.trim() || "Unknown Title",
|
||||
employer: row.employer?.trim() || "Unknown Employer",
|
||||
jobUrl,
|
||||
applicationLink: row.applicationLink?.trim() || jobUrl,
|
||||
location: row.location?.trim() || undefined,
|
||||
jobDescription: row.jobDescription?.trim() || undefined,
|
||||
isRemote: row.isRemote ?? undefined,
|
||||
};
|
||||
}
|
||||
|
||||
export async function runGoogleJobs(
|
||||
options: RunGoogleJobsOptions = {},
|
||||
): Promise<GoogleJobsResult> {
|
||||
const searchTerms =
|
||||
options.searchTerms && options.searchTerms.length > 0
|
||||
? options.searchTerms
|
||||
: ["software engineer"];
|
||||
const maxJobs = options.maxJobs ?? 30;
|
||||
|
||||
const useNpmCommand = canRunNpmCommand();
|
||||
if (!TSX_CLI_PATH && !useNpmCommand) {
|
||||
return {
|
||||
success: false,
|
||||
jobs: [],
|
||||
error: "Unable to execute Google Jobs extractor (npm/tsx unavailable)",
|
||||
};
|
||||
}
|
||||
|
||||
try {
|
||||
await new Promise<void>((resolve, reject) => {
|
||||
const extractorEnv = envForExtractorSubprocess({
|
||||
...process.env,
|
||||
GOOGLE_JOBS_SEARCH_TERMS: JSON.stringify(searchTerms),
|
||||
GOOGLE_JOBS_MAX_JOBS: String(maxJobs),
|
||||
});
|
||||
|
||||
const child = TSX_CLI_PATH
|
||||
? spawn(process.execPath, [TSX_CLI_PATH, join(srcDir, "main.ts")], {
|
||||
cwd: EXTRACTOR_DIR,
|
||||
env: extractorEnv,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
})
|
||||
: spawn("npx", ["tsx", join(srcDir, "main.ts")], {
|
||||
cwd: EXTRACTOR_DIR,
|
||||
env: extractorEnv,
|
||||
stdio: ["ignore", "pipe", "pipe"],
|
||||
});
|
||||
|
||||
let stderr = "";
|
||||
child.stderr?.on("data", (chunk: Buffer) => {
|
||||
stderr += chunk.toString();
|
||||
});
|
||||
|
||||
const timeout = setTimeout(() => {
|
||||
child.kill("SIGKILL");
|
||||
reject(new Error("Google Jobs browser scrape timed out"));
|
||||
}, SUBPROCESS_TIMEOUT_MS);
|
||||
|
||||
child.on("error", (error) => {
|
||||
clearTimeout(timeout);
|
||||
reject(error);
|
||||
});
|
||||
child.on("close", (code) => {
|
||||
clearTimeout(timeout);
|
||||
if (code === 0) {
|
||||
resolve();
|
||||
return;
|
||||
}
|
||||
reject(
|
||||
new Error(
|
||||
stderr.trim() ||
|
||||
`Google Jobs browser scrape exited with code ${code}`,
|
||||
),
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
const raw = await readFile(OUTPUT_PATH, "utf-8");
|
||||
const parsed = JSON.parse(raw) as unknown;
|
||||
const rows = Array.isArray(parsed) ? parsed : [];
|
||||
|
||||
const jobs: CreateJobInput[] = [];
|
||||
const seen = new Set<string>();
|
||||
for (const value of rows) {
|
||||
if (!value || typeof value !== "object" || Array.isArray(value)) continue;
|
||||
const mapped = mapJob(value as ScrapedJob);
|
||||
if (!mapped) continue;
|
||||
const key = mapped.jobUrl;
|
||||
if (seen.has(key)) continue;
|
||||
seen.add(key);
|
||||
jobs.push(mapped);
|
||||
}
|
||||
|
||||
return { success: true, jobs };
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : "Unknown error";
|
||||
return { success: false, jobs: [], error: message };
|
||||
}
|
||||
}
|
||||
17
extractors/google-jobs/tsconfig.json
Normal file
17
extractors/google-jobs/tsconfig.json
Normal file
@ -0,0 +1,17 @@
|
||||
{
|
||||
"compilerOptions": {
|
||||
"module": "ESNext",
|
||||
"moduleResolution": "bundler",
|
||||
"target": "ES2022",
|
||||
"outDir": "dist",
|
||||
"strict": true,
|
||||
"noUnusedLocals": false,
|
||||
"lib": ["ES2022", "DOM"],
|
||||
"types": ["node"],
|
||||
"baseUrl": ".",
|
||||
"paths": {
|
||||
"@shared/*": ["../../shared/src/*"]
|
||||
}
|
||||
},
|
||||
"include": ["./manifest.ts", "./src/**/*"]
|
||||
}
|
||||
@ -116,11 +116,32 @@ router.addHandler(
|
||||
const title = (await titleLocator.textContent())?.trim() ?? null;
|
||||
const jobUrl = toAbsolute(await titleLocator.getAttribute("href"));
|
||||
|
||||
const employerImg = article.locator("figure img");
|
||||
const employer = (await employerImg.getAttribute("alt"))?.trim() ?? null;
|
||||
|
||||
const employerAnchor = article.locator("figure a");
|
||||
const employerUrl = toAbsolute(await employerAnchor.getAttribute("href"));
|
||||
// Employer logo/link are optional — some cards omit <figure img>, and a
|
||||
// hard getAttribute timeout used to fail the whole region page (0 jobs)
|
||||
// even after we had already counted cards.
|
||||
let employer: string | null = null;
|
||||
let employerUrl: string | null = null;
|
||||
try {
|
||||
const employerImg = article.locator("figure img").first();
|
||||
if ((await employerImg.count()) > 0) {
|
||||
employer =
|
||||
(
|
||||
await employerImg.getAttribute("alt", { timeout: 2000 })
|
||||
)?.trim() ?? null;
|
||||
}
|
||||
} catch {
|
||||
// optional
|
||||
}
|
||||
try {
|
||||
const employerAnchor = article.locator("figure a").first();
|
||||
if ((await employerAnchor.count()) > 0) {
|
||||
employerUrl = toAbsolute(
|
||||
await employerAnchor.getAttribute("href", { timeout: 2000 }),
|
||||
);
|
||||
}
|
||||
} catch {
|
||||
// optional
|
||||
}
|
||||
|
||||
let disciplines: string | null = null;
|
||||
try {
|
||||
|
||||
@ -14,8 +14,10 @@ import type {
|
||||
import type { CreateJobInput } from "@shared/types/jobs";
|
||||
|
||||
const API_URL = "https://himalayas.app/jobs/api";
|
||||
const PAGE_SIZE = 50;
|
||||
const MAX_PAGES = 5;
|
||||
/** Himalayas silently caps `limit` at 20; requesting more makes us think the
|
||||
* feed ended early (`raw.length < PAGE_SIZE`) and stop after the first page. */
|
||||
const PAGE_SIZE = 20;
|
||||
const MAX_PAGES = 10;
|
||||
|
||||
interface HimalayasJob {
|
||||
title?: string;
|
||||
|
||||
@ -1,3 +1,8 @@
|
||||
# python-jobspy requires Python 3.10+ (wheels not published for 3.9 and below).
|
||||
python-jobspy
|
||||
#
|
||||
# Pinned to 1.1.82: scrape_jobs.py monkeypatches Glassdoor._fetch_jobs_page
|
||||
# to work around an upstream bug where any partial GraphQL error (even for
|
||||
# unrelated fields) discards a whole page of valid job data. If bumping this
|
||||
# version, re-check that method still matches before removing the pin.
|
||||
python-jobspy==1.1.82
|
||||
pandas
|
||||
|
||||
@ -1,12 +1,81 @@
|
||||
import csv
|
||||
import json
|
||||
import logging
|
||||
import os
|
||||
from concurrent.futures import ThreadPoolExecutor, as_completed
|
||||
from pathlib import Path
|
||||
|
||||
import pandas as pd
|
||||
from jobspy import scrape_jobs
|
||||
from jobspy.exception import GlassdoorException
|
||||
from jobspy.glassdoor import Glassdoor
|
||||
from jobspy.glassdoor.util import get_cursor_for_page
|
||||
|
||||
PROGRESS_PREFIX = "JOBOPS_PROGRESS "
|
||||
_glassdoor_log = logging.getLogger("JobSpy:Glassdoor")
|
||||
|
||||
|
||||
def _patched_glassdoor_fetch_jobs_page(
|
||||
self, scraper_input, location_id, location_type, page_num, cursor
|
||||
):
|
||||
"""Replaces python-jobspy's Glassdoor._fetch_jobs_page (as of 1.1.82).
|
||||
|
||||
Upstream treats *any* `errors` entry in Glassdoor's GraphQL response as
|
||||
fatal and discards the whole page, even when the job listings we need
|
||||
parsed fine. In practice Glassdoor's backend intermittently (but
|
||||
consistently, as of writing) returns a partial error for an unrelated
|
||||
field — observed: `jobsPageSeoData` failing with
|
||||
`SUBREQUEST_HTTP_ERROR` / "dns error" on Glassdoor's own infra — while
|
||||
`data.jobListings.jobListings` still comes back with real results. The
|
||||
upstream check throws that good data away, so every Glassdoor run
|
||||
returns 0 jobs. This patch only bails out when the job listings payload
|
||||
itself is actually missing.
|
||||
"""
|
||||
jobs = []
|
||||
self.scraper_input = scraper_input
|
||||
try:
|
||||
payload = self._add_payload(location_id, location_type, page_num, cursor)
|
||||
response = self.session.post(
|
||||
f"{self.base_url}/graph",
|
||||
timeout_seconds=15,
|
||||
data=payload,
|
||||
)
|
||||
if response.status_code != 200:
|
||||
raise GlassdoorException(
|
||||
f"bad response status code: {response.status_code}"
|
||||
)
|
||||
res_json = response.json()[0]
|
||||
job_listings = res_json.get("data", {}).get("jobListings")
|
||||
if job_listings is None:
|
||||
raise ValueError("Error encountered in API response")
|
||||
if "errors" in res_json:
|
||||
_glassdoor_log.warning(
|
||||
"Glassdoor GraphQL returned partial errors for unrelated "
|
||||
"fields; continuing with the job listings that did parse: %s",
|
||||
res_json["errors"],
|
||||
)
|
||||
except (GlassdoorException, ValueError, Exception) as e:
|
||||
_glassdoor_log.error(f"Glassdoor: {str(e)}")
|
||||
return jobs, None
|
||||
|
||||
jobs_data = job_listings["jobListings"]
|
||||
|
||||
with ThreadPoolExecutor(max_workers=self.jobs_per_page) as executor:
|
||||
future_to_job_data = {
|
||||
executor.submit(self._process_job, job): job for job in jobs_data
|
||||
}
|
||||
for future in as_completed(future_to_job_data):
|
||||
try:
|
||||
job_post = future.result()
|
||||
if job_post:
|
||||
jobs.append(job_post)
|
||||
except Exception as exc:
|
||||
raise GlassdoorException(f"Glassdoor generated an exception: {exc}")
|
||||
|
||||
return jobs, get_cursor_for_page(job_listings["paginationCursors"], page_num + 1)
|
||||
|
||||
|
||||
Glassdoor._fetch_jobs_page = _patched_glassdoor_fetch_jobs_page
|
||||
COUNTRY_ALIASES = {
|
||||
"uk": "united kingdom",
|
||||
"united kingdom": "united kingdom",
|
||||
|
||||
@ -394,15 +394,15 @@ export async function activateSearchProfileForBasicAuthUser(
|
||||
}
|
||||
if (!listPayload.ok || !Array.isArray(listPayload.data)) return;
|
||||
const normalizedUsername = username.trim().toLowerCase();
|
||||
const match = listPayload.data.find((row) => {
|
||||
const matches = listPayload.data.filter((row) => {
|
||||
const aliases = (row.data?.basicAuthUser ?? "")
|
||||
.split(",")
|
||||
.map((part) => part.trim().toLowerCase())
|
||||
.filter(Boolean);
|
||||
return aliases.includes(normalizedUsername);
|
||||
});
|
||||
if (!match) return;
|
||||
await fetch(`${API_BASE}/profiles/${match.id}/activate`, {
|
||||
if (matches.length !== 1) return;
|
||||
await fetch(`${API_BASE}/profiles/${matches[0].id}/activate`, {
|
||||
method: "POST",
|
||||
headers: {
|
||||
"Content-Type": "application/json",
|
||||
|
||||
@ -1,8 +1,17 @@
|
||||
import { PRIOR_SKIP_DISMISS_LOOKBACK_MS } from "@shared/job-fingerprint";
|
||||
import { createJob } from "@shared/testing/factories";
|
||||
import { describe, expect, it } from "vitest";
|
||||
import { buildDuplicateDismissHints } from "./job-dedup";
|
||||
|
||||
describe("buildDuplicateDismissHints", () => {
|
||||
const now = Date.parse("2026-06-10T12:00:00.000Z");
|
||||
const recentUpdatedAt = new Date(
|
||||
now - PRIOR_SKIP_DISMISS_LOOKBACK_MS + 86_400_000,
|
||||
).toISOString();
|
||||
const staleUpdatedAt = new Date(
|
||||
now - PRIOR_SKIP_DISMISS_LOOKBACK_MS - 86_400_000,
|
||||
).toISOString();
|
||||
|
||||
it("flags open jobs that match a skipped posting", () => {
|
||||
const jobs = [
|
||||
createJob({
|
||||
@ -10,6 +19,7 @@ describe("buildDuplicateDismissHints", () => {
|
||||
employer: "Acme",
|
||||
title: "SDET",
|
||||
status: "skipped",
|
||||
updatedAt: recentUpdatedAt,
|
||||
}),
|
||||
createJob({
|
||||
id: "open-1",
|
||||
@ -25,8 +35,71 @@ describe("buildDuplicateDismissHints", () => {
|
||||
}),
|
||||
];
|
||||
|
||||
const hints = buildDuplicateDismissHints(jobs);
|
||||
const hints = buildDuplicateDismissHints(jobs, now);
|
||||
expect(hints.get("open-1")).toBe("skipped");
|
||||
expect(hints.has("open-2")).toBe(false);
|
||||
});
|
||||
|
||||
it("matches employer prefix and plural title variants", () => {
|
||||
const jobs = [
|
||||
createJob({
|
||||
id: "skipped-1",
|
||||
employer: "CGI",
|
||||
title: "Automation Test Engineer",
|
||||
status: "skipped",
|
||||
updatedAt: recentUpdatedAt,
|
||||
}),
|
||||
createJob({
|
||||
id: "open-1",
|
||||
employer: ". CGI IT UK Limited",
|
||||
title: "Automation Test Engineers",
|
||||
status: "discovered",
|
||||
}),
|
||||
];
|
||||
|
||||
const hints = buildDuplicateDismissHints(jobs, now);
|
||||
expect(hints.get("open-1")).toBe("skipped");
|
||||
});
|
||||
|
||||
it("ignores skipped jobs older than the lookback window", () => {
|
||||
const jobs = [
|
||||
createJob({
|
||||
id: "skipped-1",
|
||||
employer: "Acme",
|
||||
title: "SDET",
|
||||
status: "skipped",
|
||||
updatedAt: staleUpdatedAt,
|
||||
}),
|
||||
createJob({
|
||||
id: "open-1",
|
||||
employer: "Acme",
|
||||
title: "SDET",
|
||||
status: "discovered",
|
||||
}),
|
||||
];
|
||||
|
||||
const hints = buildDuplicateDismissHints(jobs, now);
|
||||
expect(hints.has("open-1")).toBe(false);
|
||||
});
|
||||
|
||||
it("does not match different titles at the same employer", () => {
|
||||
const jobs = [
|
||||
createJob({
|
||||
id: "skipped-1",
|
||||
employer: "Cognizant",
|
||||
title: "Automation Test Engineer",
|
||||
status: "skipped",
|
||||
updatedAt: recentUpdatedAt,
|
||||
}),
|
||||
createJob({
|
||||
id: "open-1",
|
||||
employer: "Cognizant",
|
||||
title: "AI Automation Test Engineer",
|
||||
status: "discovered",
|
||||
}),
|
||||
];
|
||||
|
||||
const hints = buildDuplicateDismissHints(jobs, now);
|
||||
expect(hints.has("open-1")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
@ -1,44 +1,60 @@
|
||||
import { collectJobDedupKeys } from "@shared/job-fingerprint";
|
||||
import {
|
||||
jobsMatchEmployerAndTitleDismiss,
|
||||
PRIOR_SKIP_DISMISS_LOOKBACK_MS,
|
||||
} from "@shared/job-fingerprint";
|
||||
import type { JobListItem, JobStatus } from "@shared/types";
|
||||
|
||||
export type DuplicateDismissReason = "skipped" | "applied";
|
||||
|
||||
const OPEN_JOB_STATUSES = new Set<JobStatus>([
|
||||
"discovered",
|
||||
"ready",
|
||||
"processing",
|
||||
]);
|
||||
|
||||
function isWithinDismissLookback(updatedAt: string, nowMs: number): boolean {
|
||||
const updatedMs = Date.parse(updatedAt);
|
||||
if (!Number.isFinite(updatedMs)) return false;
|
||||
return nowMs - updatedMs <= PRIOR_SKIP_DISMISS_LOOKBACK_MS;
|
||||
}
|
||||
|
||||
/**
|
||||
* Map open jobs to a prior skip/apply when employer+title or description matches.
|
||||
* Map open jobs to a prior skip/apply when the same company and title match
|
||||
* within the last 90 days (see {@link PRIOR_SKIP_DISMISS_LOOKBACK_MS}).
|
||||
*/
|
||||
export function buildDuplicateDismissHints(
|
||||
jobs: readonly JobListItem[],
|
||||
nowMs: number = Date.now(),
|
||||
): Map<string, DuplicateDismissReason> {
|
||||
const dismissedKeys = new Map<string, DuplicateDismissReason>();
|
||||
const dismissed: Array<{
|
||||
employer: string;
|
||||
title: string;
|
||||
reason: DuplicateDismissReason;
|
||||
}> = [];
|
||||
|
||||
for (const job of jobs) {
|
||||
if (job.status !== "skipped" && job.status !== "applied") continue;
|
||||
const reason: DuplicateDismissReason =
|
||||
job.status === "applied" ? "applied" : "skipped";
|
||||
for (const key of collectJobDedupKeys({
|
||||
if (!isWithinDismissLookback(job.updatedAt, nowMs)) continue;
|
||||
dismissed.push({
|
||||
employer: job.employer,
|
||||
title: job.title,
|
||||
})) {
|
||||
if (!dismissedKeys.has(key)) dismissedKeys.set(key, reason);
|
||||
}
|
||||
reason: job.status === "applied" ? "applied" : "skipped",
|
||||
});
|
||||
}
|
||||
|
||||
const hints = new Map<string, DuplicateDismissReason>();
|
||||
const openStatuses = new Set<JobStatus>([
|
||||
"discovered",
|
||||
"ready",
|
||||
"processing",
|
||||
]);
|
||||
|
||||
for (const job of jobs) {
|
||||
if (!openStatuses.has(job.status)) continue;
|
||||
for (const key of collectJobDedupKeys({
|
||||
employer: job.employer,
|
||||
title: job.title,
|
||||
})) {
|
||||
const reason = dismissedKeys.get(key);
|
||||
if (reason) {
|
||||
hints.set(job.id, reason);
|
||||
if (!OPEN_JOB_STATUSES.has(job.status)) continue;
|
||||
for (const prior of dismissed) {
|
||||
if (
|
||||
jobsMatchEmployerAndTitleDismiss({
|
||||
employerA: job.employer,
|
||||
titleA: job.title,
|
||||
employerB: prior.employer,
|
||||
titleB: prior.title,
|
||||
})
|
||||
) {
|
||||
hints.set(job.id, prior.reason);
|
||||
break;
|
||||
}
|
||||
}
|
||||
@ -46,5 +62,3 @@ export function buildDuplicateDismissHints(
|
||||
|
||||
return hints;
|
||||
}
|
||||
|
||||
export { collectJobDedupKeys };
|
||||
|
||||
@ -63,6 +63,8 @@ export const OrchestratorPage: React.FC = () => {
|
||||
setEmployerFilterTokens,
|
||||
applySettingsCompanySkipList,
|
||||
setApplySettingsCompanySkipList,
|
||||
hidePriorSkips,
|
||||
setHidePriorSkips,
|
||||
sort,
|
||||
setSort,
|
||||
resetFilters,
|
||||
@ -173,8 +175,8 @@ export const OrchestratorPage: React.FC = () => {
|
||||
);
|
||||
|
||||
const duplicateDismissHints = useMemo(
|
||||
() => buildDuplicateDismissHints(jobs),
|
||||
[jobs],
|
||||
() => (hidePriorSkips ? buildDuplicateDismissHints(jobs) : undefined),
|
||||
[hidePriorSkips, jobs],
|
||||
);
|
||||
|
||||
const jobListFilterExtras = useMemo(
|
||||
@ -531,6 +533,8 @@ export const OrchestratorPage: React.FC = () => {
|
||||
onApplySettingsCompanySkipListChange={
|
||||
setApplySettingsCompanySkipList
|
||||
}
|
||||
hidePriorSkips={hidePriorSkips}
|
||||
onHidePriorSkipsChange={setHidePriorSkips}
|
||||
sponsorFilter={sponsorFilter}
|
||||
onSponsorFilterChange={setSponsorFilter}
|
||||
sponsorshipSignalsFilter={sponsorshipSignalsFilter}
|
||||
|
||||
@ -640,6 +640,7 @@ export const AutomaticRunTab: React.FC<AutomaticRunTabProps> = ({
|
||||
</CardHeader>
|
||||
<CardContent className="space-y-4">
|
||||
<KeywordSetPicker
|
||||
key={settings?.activeProfileId ?? "default"}
|
||||
disabled={isSaving || isPipelineRunning}
|
||||
onActiveSetChange={handleActiveKeywordSetChange}
|
||||
/>
|
||||
|
||||
@ -53,6 +53,8 @@ const renderFilters = (overrides?: Partial<FiltersProps>) => {
|
||||
onEmployerIncludeChange: vi.fn(),
|
||||
onEmployerExcludeChange: vi.fn(),
|
||||
applySettingsCompanySkipList: true,
|
||||
hidePriorSkips: true,
|
||||
onHidePriorSkipsChange: vi.fn(),
|
||||
onApplySettingsCompanySkipListChange: vi.fn(),
|
||||
sponsorFilter: "all" as SponsorFilter,
|
||||
onSponsorFilterChange: vi.fn(),
|
||||
|
||||
@ -69,6 +69,8 @@ interface OrchestratorFiltersProps {
|
||||
onEmployerExcludeChange: (values: string[]) => void;
|
||||
applySettingsCompanySkipList: boolean;
|
||||
onApplySettingsCompanySkipListChange: (value: boolean) => void;
|
||||
hidePriorSkips: boolean;
|
||||
onHidePriorSkipsChange: (value: boolean) => void;
|
||||
sponsorFilter: SponsorFilter;
|
||||
onSponsorFilterChange: (value: SponsorFilter) => void;
|
||||
sponsorshipSignalsFilter: SponsorshipSignalId[];
|
||||
@ -183,6 +185,8 @@ export const OrchestratorFilters: React.FC<OrchestratorFiltersProps> = ({
|
||||
onEmployerExcludeChange,
|
||||
applySettingsCompanySkipList,
|
||||
onApplySettingsCompanySkipListChange,
|
||||
hidePriorSkips,
|
||||
onHidePriorSkipsChange,
|
||||
sponsorFilter,
|
||||
onSponsorFilterChange,
|
||||
sponsorshipSignalsFilter,
|
||||
@ -253,6 +257,7 @@ export const OrchestratorFilters: React.FC<OrchestratorFiltersProps> = ({
|
||||
employerIncludeFilter.length > 0 || employerExcludeFilter.length > 0,
|
||||
) +
|
||||
Number(!applySettingsCompanySkipList) +
|
||||
Number(!hidePriorSkips) +
|
||||
Number(sponsorFilter !== "all") +
|
||||
Number(sponsorshipSignalsFilter.length > 0) +
|
||||
Number(hideSponsorBlockers) +
|
||||
@ -271,6 +276,7 @@ export const OrchestratorFilters: React.FC<OrchestratorFiltersProps> = ({
|
||||
employerIncludeFilter.length,
|
||||
employerExcludeFilter.length,
|
||||
applySettingsCompanySkipList,
|
||||
hidePriorSkips,
|
||||
sponsorFilter,
|
||||
sponsorshipSignalsFilter.length,
|
||||
hideSponsorBlockers,
|
||||
@ -542,6 +548,29 @@ export const OrchestratorFilters: React.FC<OrchestratorFiltersProps> = ({
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<div className="flex items-start gap-3 rounded-md border border-border/60 bg-muted/20 px-3 py-3">
|
||||
<Checkbox
|
||||
id="hide-prior-skips"
|
||||
checked={hidePriorSkips}
|
||||
onCheckedChange={(checked) => {
|
||||
onHidePriorSkipsChange(checked === true);
|
||||
}}
|
||||
/>
|
||||
<div className="space-y-1">
|
||||
<label
|
||||
htmlFor="hide-prior-skips"
|
||||
className="text-sm font-medium leading-none cursor-pointer"
|
||||
>
|
||||
Hide roles you already skipped
|
||||
</label>
|
||||
<p className="text-xs text-muted-foreground">
|
||||
Hides Discovered and Ready rows that match a skipped
|
||||
or applied job with the same company and title from
|
||||
the last 90 days.
|
||||
</p>
|
||||
</div>
|
||||
</div>
|
||||
|
||||
<TokenizedInput
|
||||
id="employer-include-filter"
|
||||
values={employerIncludeFilter}
|
||||
|
||||
@ -5,7 +5,7 @@ import type { SearchProfile } from "@shared/types";
|
||||
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import { Loader2 } from "lucide-react";
|
||||
import type React from "react";
|
||||
import { useMemo } from "react";
|
||||
import { useEffect, useMemo, useState } from "react";
|
||||
import { toast } from "sonner";
|
||||
import { Button } from "@/components/ui/button";
|
||||
import {
|
||||
@ -27,17 +27,38 @@ export const ProfileQuickSwitch: React.FC = () => {
|
||||
const { data: profiles = [], isLoading } = useQuery<SearchProfile[]>({
|
||||
queryKey: queryKeys.searchProfiles.list(),
|
||||
queryFn: api.listProfiles,
|
||||
select: (rows) => {
|
||||
const seen = new Set<string>();
|
||||
return rows.filter((profile) => {
|
||||
if (seen.has(profile.id)) return false;
|
||||
seen.add(profile.id);
|
||||
return true;
|
||||
});
|
||||
},
|
||||
});
|
||||
|
||||
const [selectedId, setSelectedId] = useState<string | null>(null);
|
||||
|
||||
const activeId = useMemo(() => {
|
||||
if (selectedId && profiles.some((profile) => profile.id === selectedId)) {
|
||||
return selectedId;
|
||||
}
|
||||
const sid = settings?.activeProfileId?.trim() ?? "";
|
||||
if (sid && profiles.some((p) => p.id === sid)) return sid;
|
||||
if (sid && profiles.some((profile) => profile.id === sid)) return sid;
|
||||
return profiles[0]?.id ?? "";
|
||||
}, [selectedId, settings?.activeProfileId, profiles]);
|
||||
|
||||
useEffect(() => {
|
||||
const sid = settings?.activeProfileId?.trim() ?? "";
|
||||
if (sid && profiles.some((profile) => profile.id === sid)) {
|
||||
setSelectedId(sid);
|
||||
}
|
||||
}, [settings?.activeProfileId, profiles]);
|
||||
|
||||
const activateMutation = useMutation({
|
||||
mutationFn: (id: string) => api.activateProfile(id),
|
||||
onSuccess: async () => {
|
||||
onSuccess: async (_data, profileId) => {
|
||||
setSelectedId(profileId);
|
||||
await Promise.all([
|
||||
queryClient.invalidateQueries({ queryKey: queryKeys.jobs.all }),
|
||||
queryClient.invalidateQueries({ queryKey: queryKeys.profile.all }),
|
||||
@ -46,7 +67,10 @@ export const ProfileQuickSwitch: React.FC = () => {
|
||||
}),
|
||||
]);
|
||||
await refreshSettings();
|
||||
toast.success("Search profile updated");
|
||||
const profileName =
|
||||
profiles.find((profile) => profile.id === profileId)?.name ??
|
||||
"Search profile";
|
||||
toast.success(`${profileName} active`);
|
||||
},
|
||||
onError: (err: unknown) => {
|
||||
const message =
|
||||
|
||||
@ -199,6 +199,7 @@ export function buildMaxCoverageDiscoveryLimits(args: {
|
||||
builtinMaxJobsPerTerm: boardCap,
|
||||
builtinMaxPagesPerTerm: 10,
|
||||
wellfoundMaxJobsPerTerm: Math.min(100, boardCap),
|
||||
googleJobsMaxJobsPerTerm: Math.min(30, boardCap),
|
||||
qajobsboardMaxJobsPerTerm: boardCap,
|
||||
arcMaxJobsPerPath: boardCap,
|
||||
smartrecruitersMaxJobsPerCompany: boardCap,
|
||||
|
||||
@ -435,6 +435,11 @@ export const useOrchestratorFilters = () => {
|
||||
[searchParams],
|
||||
);
|
||||
|
||||
const hidePriorSkips = useMemo(
|
||||
() => searchParams.get("hidePriorSkips") !== "0",
|
||||
[searchParams],
|
||||
);
|
||||
|
||||
const setApplySettingsCompanySkipList = useCallback(
|
||||
(value: boolean) => {
|
||||
setSearchParams(
|
||||
@ -449,6 +454,20 @@ export const useOrchestratorFilters = () => {
|
||||
[setSearchParams],
|
||||
);
|
||||
|
||||
const setHidePriorSkips = useCallback(
|
||||
(value: boolean) => {
|
||||
setSearchParams(
|
||||
(prev) => {
|
||||
if (value) prev.delete("hidePriorSkips");
|
||||
else prev.set("hidePriorSkips", "0");
|
||||
return prev;
|
||||
},
|
||||
{ replace: true },
|
||||
);
|
||||
},
|
||||
[setSearchParams],
|
||||
);
|
||||
|
||||
const sort = useMemo((): JobSort => {
|
||||
const sortValue = searchParams.get("sort");
|
||||
if (!sortValue) return DEFAULT_SORT;
|
||||
@ -508,6 +527,7 @@ export const useOrchestratorFilters = () => {
|
||||
prev.delete("employer");
|
||||
prev.delete("employerExclude");
|
||||
prev.delete("skipList");
|
||||
prev.delete("hidePriorSkips");
|
||||
return prev;
|
||||
},
|
||||
{ replace: true },
|
||||
@ -540,6 +560,8 @@ export const useOrchestratorFilters = () => {
|
||||
setEmployerFilterTokens,
|
||||
applySettingsCompanySkipList,
|
||||
setApplySettingsCompanySkipList,
|
||||
hidePriorSkips,
|
||||
setHidePriorSkips,
|
||||
sort,
|
||||
setSort,
|
||||
resetFilters,
|
||||
|
||||
@ -34,6 +34,7 @@ describe("orchestrator utils", () => {
|
||||
"testdevjobs",
|
||||
"wellfound",
|
||||
"builtin",
|
||||
"google-jobs",
|
||||
] as const) {
|
||||
expect(enabled).toContain(source);
|
||||
}
|
||||
|
||||
@ -334,6 +334,7 @@ export const getEnabledSources = (
|
||||
source === "testdevjobs" ||
|
||||
source === "wellfound" ||
|
||||
source === "builtin" ||
|
||||
source === "google-jobs" ||
|
||||
source === "qajobsboard" ||
|
||||
source === "arcdev"
|
||||
) {
|
||||
|
||||
@ -104,8 +104,8 @@ export const JobSourcesSettingsSection: React.FC<
|
||||
<p className="text-sm text-muted-foreground">
|
||||
Configure company slugs, tenant IDs, and career-page URLs so ATS and
|
||||
direct employer sources appear in the pipeline run dialog. Public job
|
||||
boards (Working Nomads, TestDevJobs, Wellfound, etc.) need no setup
|
||||
here.
|
||||
boards (Working Nomads, TestDevJobs, Wellfound, Google Jobs, etc.)
|
||||
need no setup here.
|
||||
</p>
|
||||
|
||||
<div className="space-y-4">
|
||||
|
||||
@ -175,6 +175,10 @@ profilesRouter.post(
|
||||
const basicUser = parseBasicAuthUsername(req.headers.authorization)?.trim();
|
||||
|
||||
if (isBasicAuthEnabled() && basicUser) {
|
||||
await profilesRepo.setActiveSearchProfileIdForBasicAuthUser(
|
||||
basicUser,
|
||||
profile.id,
|
||||
);
|
||||
clearProfileCache();
|
||||
return ok(res, { activated: true, profileId: profile.id });
|
||||
}
|
||||
|
||||
@ -271,6 +271,7 @@ export const DEMO_SOURCE_BASE_URLS: Record<JobSource, string> = {
|
||||
testdevjobs: "https://testdevjobs.com",
|
||||
wellfound: "https://wellfound.com",
|
||||
builtin: "https://builtin.com",
|
||||
"google-jobs": "https://www.google.com/search?ibp=htl;jobs",
|
||||
fourdayweek: "https://4dayweek.io",
|
||||
ashby: "https://jobs.ashbyhq.com",
|
||||
lever: "https://jobs.lever.co",
|
||||
|
||||
@ -29,6 +29,75 @@ function sqlInsertSearchProfileSeed(input: {
|
||||
return `INSERT OR IGNORE INTO search_profiles (id, name, data, created_at, updated_at) VALUES ('${id}', '${name}', '${data}', datetime('now'), datetime('now'))`;
|
||||
}
|
||||
|
||||
function sqlJsonLiteral(values: string[]): string {
|
||||
return JSON.stringify(values).replace(/'/g, "''");
|
||||
}
|
||||
|
||||
/** Canada / remote — SDET & QA automation title variants for discovery + scoring. */
|
||||
const ILIA_DOBKIN_CA_TARGET_ROLES = [
|
||||
"SDET",
|
||||
"Senior SDET",
|
||||
"Lead SDET",
|
||||
"Staff SDET",
|
||||
"Principal SDET",
|
||||
"Software Development Engineer in Test",
|
||||
"Senior Software Development Engineer in Test",
|
||||
"Software Engineer in Test",
|
||||
"Software Test Engineer",
|
||||
"Senior Software Test Engineer",
|
||||
"QA Automation Engineer",
|
||||
"Senior QA Automation Engineer",
|
||||
"Quality Assurance Automation Engineer",
|
||||
"Test Automation Engineer",
|
||||
"Senior Test Automation Engineer",
|
||||
"Automation Test Engineer",
|
||||
"Automation Engineer",
|
||||
"QA Engineer",
|
||||
"Senior QA Engineer",
|
||||
"Quality Engineer",
|
||||
"Test Automation Lead",
|
||||
"QA Automation Lead",
|
||||
"Test Lead",
|
||||
];
|
||||
|
||||
/** Job-board search terms — title variants plus common stack/tool pairings. */
|
||||
const ILIA_DOBKIN_CA_KEYWORD_TERMS = [
|
||||
...ILIA_DOBKIN_CA_TARGET_ROLES.map((role) => role.toLowerCase()),
|
||||
"playwright SDET",
|
||||
"SDET playwright",
|
||||
"SDET java",
|
||||
"SDET typescript",
|
||||
"SDET cypress",
|
||||
"SDET selenium",
|
||||
"SDET api testing",
|
||||
"SDET remote",
|
||||
"QA automation remote",
|
||||
"contract SDET",
|
||||
"contract QA automation",
|
||||
"SET",
|
||||
"software engineer test",
|
||||
];
|
||||
|
||||
const ILIA_DOBKIN_US_TARGET_ROLES = [
|
||||
...ILIA_DOBKIN_CA_TARGET_ROLES,
|
||||
"Computer Systems Analyst",
|
||||
"Quality Assurance Engineer",
|
||||
];
|
||||
|
||||
const ILIA_DOBKIN_US_KEYWORD_TERMS = [
|
||||
...ILIA_DOBKIN_CA_KEYWORD_TERMS,
|
||||
"computer systems analyst",
|
||||
"systems analyst",
|
||||
"computer systems analyst QA",
|
||||
"SDET United States",
|
||||
"SDET USA",
|
||||
"SDET remote US",
|
||||
"QA automation engineer USA",
|
||||
"test automation engineer United States",
|
||||
"quality assurance engineer automation",
|
||||
"TN SDET",
|
||||
];
|
||||
|
||||
const migrations = [
|
||||
`CREATE TABLE IF NOT EXISTS jobs (
|
||||
id TEXT PRIMARY KEY,
|
||||
@ -817,28 +886,40 @@ const migrations = [
|
||||
id: "685b0000-0000-4000-8000-000000000001",
|
||||
name: "Ilia Dobkin",
|
||||
profile: {
|
||||
targetRoles: [
|
||||
"Software Development Engineer in Test",
|
||||
"QA Automation Engineer",
|
||||
"SDET",
|
||||
],
|
||||
targetRoles: ILIA_DOBKIN_CA_TARGET_ROLES,
|
||||
experienceLevel: "Senior",
|
||||
mustHaveSkills: [
|
||||
"Playwright",
|
||||
"Cypress",
|
||||
"Selenium",
|
||||
"TypeScript",
|
||||
"Java",
|
||||
"API testing",
|
||||
"CI/CD",
|
||||
"Selenium",
|
||||
"Cypress",
|
||||
],
|
||||
niceToHaveSkills: [
|
||||
"Spring Boot",
|
||||
"Python",
|
||||
".NET",
|
||||
"Contract testing",
|
||||
"PostgreSQL",
|
||||
"JUnit",
|
||||
],
|
||||
niceToHaveSkills: ["Python", ".NET", "CI/CD"],
|
||||
dealBreakers: [],
|
||||
preferredWorkArrangement: ["remote", "hybrid"],
|
||||
preferredLocations: ["Toronto", "Ontario", "Canada"],
|
||||
preferredLocations: [
|
||||
"Toronto",
|
||||
"Ontario",
|
||||
"GTA",
|
||||
"Greater Toronto Area",
|
||||
"Canada",
|
||||
"Remote Canada",
|
||||
],
|
||||
minimumSalary: "",
|
||||
industriesToTarget: ["Software", "FinTech", "iGaming"],
|
||||
industriesToTarget: ["Software", "FinTech", "iGaming", "Enterprise SaaS"],
|
||||
industriesToAvoid: [],
|
||||
aboutMe:
|
||||
"Ilia Dobkin — SDET / test automation; paired login user `ilia` activates this profile and local resume ilia-dobkin.json (unset JOBOPS_LOCAL_RESUME_PATH so Settings path applies).",
|
||||
"Ilia Dobkin — Senior SDET / test automation (Canada & remote). Paired login user `ilia` activates this profile and local resume ilia-dobkin.json (unset JOBOPS_LOCAL_RESUME_PATH so Settings path applies).",
|
||||
basicAuthUser: "ilia,dobkin",
|
||||
resumeLocalPath: "../data/resumes/ilia-dobkin.json",
|
||||
},
|
||||
@ -867,7 +948,7 @@ const migrations = [
|
||||
'ilia-dobkin-keyword-set',
|
||||
'685b0000-0000-4000-8000-000000000001',
|
||||
'SDET',
|
||||
'["SDET","QA automation engineer","automation test engineer","playwright","software development engineer in test","quality assurance automation engineer"]',
|
||||
'${sqlJsonLiteral(ILIA_DOBKIN_CA_KEYWORD_TERMS)}',
|
||||
1,
|
||||
datetime('now'),
|
||||
datetime('now')
|
||||
@ -976,6 +1057,91 @@ const migrations = [
|
||||
updated_at = datetime('now')
|
||||
WHERE id = '685b0000-0000-4000-8000-000000000002'
|
||||
AND coalesce(json_extract(data, '$.resumeLocalPath'), '') = ''`,
|
||||
sqlInsertSearchProfileSeed({
|
||||
id: "685b0000-0000-4000-8000-000000000003",
|
||||
name: "Ilia Dobkin (US)",
|
||||
profile: {
|
||||
targetRoles: ILIA_DOBKIN_US_TARGET_ROLES,
|
||||
experienceLevel: "Senior",
|
||||
mustHaveSkills: [
|
||||
"Playwright",
|
||||
"TypeScript",
|
||||
"API testing",
|
||||
"CI/CD",
|
||||
"Java",
|
||||
"Integration testing",
|
||||
"Selenium",
|
||||
],
|
||||
niceToHaveSkills: [
|
||||
"Cypress",
|
||||
"Spring Boot",
|
||||
"Python",
|
||||
".NET",
|
||||
"Contract testing",
|
||||
"PostgreSQL",
|
||||
],
|
||||
dealBreakers: [],
|
||||
preferredWorkArrangement: ["remote", "hybrid"],
|
||||
preferredLocations: ["United States", "U.S.", "USA", "Remote US", "TN"],
|
||||
minimumSalary: "",
|
||||
industriesToTarget: [
|
||||
"Software",
|
||||
"FinTech",
|
||||
"Financial services",
|
||||
"Enterprise SaaS",
|
||||
],
|
||||
industriesToAvoid: [],
|
||||
aboutMe:
|
||||
"Ilia Dobkin (US) — Senior SDET / QA automation; Canadian citizen, TN-eligible (USMCA), open to U.S. relocation. Computer Systems Analyst-aligned framing. Switch to this profile in the Jobs page picker (same login as Canada).",
|
||||
basicAuthUser: "ilia,dobkin",
|
||||
resumeLocalPath: "../data/resumes/ilia-dobkin-us.json",
|
||||
},
|
||||
}),
|
||||
`UPDATE search_profiles
|
||||
SET data = json_set(data, '$.targetRoles', '${sqlJsonLiteral(ILIA_DOBKIN_CA_TARGET_ROLES)}'),
|
||||
updated_at = datetime('now')
|
||||
WHERE id = '685b0000-0000-4000-8000-000000000001'`,
|
||||
`UPDATE keyword_sets
|
||||
SET terms = '${sqlJsonLiteral(ILIA_DOBKIN_CA_KEYWORD_TERMS)}',
|
||||
updated_at = datetime('now')
|
||||
WHERE id = 'ilia-dobkin-keyword-set'`,
|
||||
`INSERT OR IGNORE INTO keyword_sets (id, owner_profile_id, name, terms, is_active, created_at, updated_at)
|
||||
VALUES (
|
||||
'ilia-dobkin-us-keyword-set',
|
||||
'685b0000-0000-4000-8000-000000000003',
|
||||
'SDET',
|
||||
'${sqlJsonLiteral(ILIA_DOBKIN_US_KEYWORD_TERMS)}',
|
||||
1,
|
||||
datetime('now'),
|
||||
datetime('now')
|
||||
)`,
|
||||
`UPDATE search_profiles
|
||||
SET data = json_set(
|
||||
json_set(data, '$.basicAuthUser', 'ilia,dobkin'),
|
||||
'$.aboutMe',
|
||||
'Ilia Dobkin (US) — Senior SDET / QA automation; Canadian citizen, TN-eligible (USMCA), open to U.S. relocation. Computer Systems Analyst-aligned framing. Switch to this profile in the Jobs page picker (same login as Canada).'
|
||||
),
|
||||
updated_at = datetime('now')
|
||||
WHERE id = '685b0000-0000-4000-8000-000000000003'`,
|
||||
`UPDATE search_profiles
|
||||
SET data = json_set(data, '$.targetRoles', '${sqlJsonLiteral(ILIA_DOBKIN_US_TARGET_ROLES)}'),
|
||||
updated_at = datetime('now')
|
||||
WHERE id = '685b0000-0000-4000-8000-000000000003'`,
|
||||
`UPDATE keyword_sets
|
||||
SET terms = '${sqlJsonLiteral(ILIA_DOBKIN_US_KEYWORD_TERMS)}',
|
||||
updated_at = datetime('now')
|
||||
WHERE id = 'ilia-dobkin-us-keyword-set'`,
|
||||
// Remove duplicate Ilia profiles left by repository tests (same names, test UUIDs).
|
||||
`DELETE FROM keyword_sets
|
||||
WHERE owner_profile_id IN (
|
||||
'685b0000-0000-4000-8000-000000009901',
|
||||
'685b0000-0000-4000-8000-000000009902'
|
||||
)`,
|
||||
`DELETE FROM search_profiles
|
||||
WHERE id IN (
|
||||
'685b0000-0000-4000-8000-000000009901',
|
||||
'685b0000-0000-4000-8000-000000009902'
|
||||
)`,
|
||||
];
|
||||
|
||||
console.log("🔧 Running database migrations...");
|
||||
|
||||
@ -50,6 +50,9 @@ async function loadJobDedupIndexes(ownerProfileId: string): Promise<{
|
||||
existingContentFingerprintSet: Set<string>;
|
||||
dismissedDedupKeySet: Set<string>;
|
||||
}> {
|
||||
const dismissLookbackMs = 90 * 24 * 60 * 60 * 1000;
|
||||
const dismissCutoffMs = Date.now() - dismissLookbackMs;
|
||||
|
||||
const rows = await db
|
||||
.select({
|
||||
jobUrl: jobs.jobUrl,
|
||||
@ -60,6 +63,7 @@ async function loadJobDedupIndexes(ownerProfileId: string): Promise<{
|
||||
title: jobs.title,
|
||||
jobDescription: jobs.jobDescription,
|
||||
status: jobs.status,
|
||||
updatedAt: jobs.updatedAt,
|
||||
})
|
||||
.from(jobs)
|
||||
.where(eq(jobs.ownerProfileId, ownerProfileId));
|
||||
@ -94,12 +98,15 @@ async function loadJobDedupIndexes(ownerProfileId: string): Promise<{
|
||||
}
|
||||
|
||||
if (row.status === "skipped" || row.status === "applied") {
|
||||
for (const key of collectJobDedupKeys({
|
||||
employer: row.employer,
|
||||
title: row.title,
|
||||
jobDescription: row.jobDescription,
|
||||
})) {
|
||||
dismissedDedupKeySet.add(key);
|
||||
const updatedMs = Date.parse(row.updatedAt);
|
||||
if (Number.isFinite(updatedMs) && updatedMs >= dismissCutoffMs) {
|
||||
for (const key of collectJobDedupKeys({
|
||||
employer: row.employer,
|
||||
title: row.title,
|
||||
jobDescription: row.jobDescription,
|
||||
})) {
|
||||
dismissedDedupKeySet.add(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
108
orchestrator/src/server/repositories/profiles.test.ts
Normal file
108
orchestrator/src/server/repositories/profiles.test.ts
Normal file
@ -0,0 +1,108 @@
|
||||
import { eq, inArray } from "drizzle-orm";
|
||||
import { afterEach, beforeEach, describe, expect, it } from "vitest";
|
||||
import { db } from "../db";
|
||||
import { searchProfiles } from "../db/schema";
|
||||
import {
|
||||
getSearchProfileIdForBasicAuthUser,
|
||||
listProfilesForBasicAuthUser,
|
||||
profileMatchesBasicAuthUser,
|
||||
setActiveSearchProfileIdForBasicAuthUser,
|
||||
} from "./profiles";
|
||||
|
||||
const CA_PROFILE_ID = "685b0000-0000-4000-8000-000000009901";
|
||||
const US_PROFILE_ID = "685b0000-0000-4000-8000-000000009902";
|
||||
|
||||
const sharedProfileData = {
|
||||
targetRoles: ["SDET"],
|
||||
experienceLevel: "Senior",
|
||||
mustHaveSkills: [],
|
||||
niceToHaveSkills: [],
|
||||
dealBreakers: [],
|
||||
preferredWorkArrangement: [],
|
||||
preferredLocations: [],
|
||||
minimumSalary: "",
|
||||
industriesToTarget: [],
|
||||
industriesToAvoid: [],
|
||||
aboutMe: "",
|
||||
basicAuthUser: "__test_profiles_repo__",
|
||||
};
|
||||
|
||||
describe("profiles repository", () => {
|
||||
afterEach(async () => {
|
||||
await db
|
||||
.delete(searchProfiles)
|
||||
.where(inArray(searchProfiles.id, [CA_PROFILE_ID, US_PROFILE_ID]));
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await db
|
||||
.delete(searchProfiles)
|
||||
.where(inArray(searchProfiles.id, [CA_PROFILE_ID, US_PROFILE_ID]));
|
||||
await db.insert(searchProfiles).values([
|
||||
{
|
||||
id: CA_PROFILE_ID,
|
||||
name: "Test Profiles Repo CA",
|
||||
data: JSON.stringify({
|
||||
...sharedProfileData,
|
||||
preferredLocations: ["Canada"],
|
||||
resumeLocalPath: "../data/resumes/ilia-dobkin.json",
|
||||
}),
|
||||
},
|
||||
{
|
||||
id: US_PROFILE_ID,
|
||||
name: "Test Profiles Repo US",
|
||||
data: JSON.stringify({
|
||||
...sharedProfileData,
|
||||
preferredLocations: ["United States"],
|
||||
resumeLocalPath: "../data/resumes/ilia-dobkin-us.json",
|
||||
}),
|
||||
},
|
||||
]);
|
||||
});
|
||||
|
||||
it("matches comma-separated basic auth aliases", () => {
|
||||
expect(profileMatchesBasicAuthUser("ilia,dobkin", "ILIA")).toBe(true);
|
||||
expect(profileMatchesBasicAuthUser("ilia,dobkin", "dobkin")).toBe(true);
|
||||
expect(profileMatchesBasicAuthUser("ilia,dobkin", "cherepaha")).toBe(false);
|
||||
});
|
||||
|
||||
it("lists every profile visible to the login", async () => {
|
||||
const visible = await listProfilesForBasicAuthUser(
|
||||
"__test_profiles_repo__",
|
||||
);
|
||||
expect(visible.map((profile) => profile.id).sort()).toEqual(
|
||||
[CA_PROFILE_ID, US_PROFILE_ID].sort(),
|
||||
);
|
||||
});
|
||||
|
||||
it("defaults to the first matching profile when none is stored", async () => {
|
||||
await expect(
|
||||
getSearchProfileIdForBasicAuthUser("__test_profiles_repo__"),
|
||||
).resolves.toBe(CA_PROFILE_ID);
|
||||
});
|
||||
|
||||
it("remembers the active profile across basic auth aliases", async () => {
|
||||
await db
|
||||
.update(searchProfiles)
|
||||
.set({
|
||||
data: JSON.stringify({
|
||||
...sharedProfileData,
|
||||
preferredLocations: ["Canada"],
|
||||
basicAuthUser: "test-repo-a,test-repo-b",
|
||||
}),
|
||||
})
|
||||
.where(eq(searchProfiles.id, US_PROFILE_ID));
|
||||
|
||||
await setActiveSearchProfileIdForBasicAuthUser(
|
||||
"test-repo-a",
|
||||
US_PROFILE_ID,
|
||||
);
|
||||
|
||||
await expect(
|
||||
getSearchProfileIdForBasicAuthUser("test-repo-a"),
|
||||
).resolves.toBe(US_PROFILE_ID);
|
||||
await expect(
|
||||
getSearchProfileIdForBasicAuthUser("test-repo-b"),
|
||||
).resolves.toBe(US_PROFILE_ID);
|
||||
});
|
||||
});
|
||||
@ -8,6 +8,13 @@ import type {
|
||||
import { eq } from "drizzle-orm";
|
||||
import { db } from "../db";
|
||||
import { searchProfiles } from "../db/schema";
|
||||
import type { SettingKey } from "./settings";
|
||||
import { getSetting, setSetting } from "./settings";
|
||||
|
||||
const BASIC_AUTH_ACTIVE_PROFILE_BY_USER_KEY: SettingKey =
|
||||
"basicAuthActiveProfileByUser";
|
||||
|
||||
type BasicAuthActiveProfileByUser = Record<string, string>;
|
||||
|
||||
function mapRow(row: typeof searchProfiles.$inferSelect): SearchProfile {
|
||||
let data: JobSearchProfile;
|
||||
@ -110,16 +117,76 @@ export async function updateProfile(
|
||||
return getProfileById(id);
|
||||
}
|
||||
|
||||
function parseBasicAuthActiveProfileByUser(
|
||||
raw: string | null | undefined,
|
||||
): BasicAuthActiveProfileByUser {
|
||||
if (!raw?.trim()) return {};
|
||||
try {
|
||||
const parsed: unknown = JSON.parse(raw);
|
||||
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
|
||||
return {};
|
||||
}
|
||||
const out: BasicAuthActiveProfileByUser = {};
|
||||
for (const [key, value] of Object.entries(parsed)) {
|
||||
const alias = key.trim().toLowerCase();
|
||||
const profileId = typeof value === "string" ? value.trim() : "";
|
||||
if (alias && profileId) out[alias] = profileId;
|
||||
}
|
||||
return out;
|
||||
} catch {
|
||||
return {};
|
||||
}
|
||||
}
|
||||
|
||||
async function loadBasicAuthActiveProfileByUser(): Promise<BasicAuthActiveProfileByUser> {
|
||||
const raw = await getSetting(BASIC_AUTH_ACTIVE_PROFILE_BY_USER_KEY);
|
||||
return parseBasicAuthActiveProfileByUser(raw);
|
||||
}
|
||||
|
||||
function basicAuthAliases(basicAuthUser: string | null | undefined): string[] {
|
||||
return (basicAuthUser ?? "")
|
||||
.split(",")
|
||||
.map((part) => part.trim().toLowerCase())
|
||||
.filter(Boolean);
|
||||
}
|
||||
|
||||
/**
|
||||
* When one login maps to multiple search profiles, remember which profile is active.
|
||||
* Writes the same profile id for every alias on that profile (e.g. ilia + dobkin).
|
||||
*/
|
||||
export async function setActiveSearchProfileIdForBasicAuthUser(
|
||||
username: string,
|
||||
profileId: string,
|
||||
): Promise<void> {
|
||||
const profile = await getProfileById(profileId);
|
||||
if (!profile) return;
|
||||
|
||||
const requestAlias = username.trim().toLowerCase();
|
||||
if (!requestAlias) return;
|
||||
if (!profileMatchesBasicAuthUser(profile.data.basicAuthUser, requestAlias)) {
|
||||
return;
|
||||
}
|
||||
|
||||
const map = await loadBasicAuthActiveProfileByUser();
|
||||
for (const alias of basicAuthAliases(profile.data.basicAuthUser)) {
|
||||
map[alias] = profileId;
|
||||
}
|
||||
await setSetting(BASIC_AUTH_ACTIVE_PROFILE_BY_USER_KEY, JSON.stringify(map));
|
||||
}
|
||||
|
||||
export async function getSearchProfileIdForBasicAuthUser(
|
||||
username: string,
|
||||
): Promise<string | null> {
|
||||
const profiles = await listProfiles();
|
||||
for (const p of profiles) {
|
||||
if (profileMatchesBasicAuthUser(p.data.basicAuthUser, username)) {
|
||||
return p.id;
|
||||
}
|
||||
const matches = await listProfilesForBasicAuthUser(username);
|
||||
if (matches.length === 0) return null;
|
||||
if (matches.length === 1) return matches[0].id;
|
||||
|
||||
const map = await loadBasicAuthActiveProfileByUser();
|
||||
const preferred = map[username.trim().toLowerCase()];
|
||||
if (preferred && matches.some((profile) => profile.id === preferred)) {
|
||||
return preferred;
|
||||
}
|
||||
return null;
|
||||
return matches[0].id;
|
||||
}
|
||||
|
||||
export async function deleteProfile(id: string): Promise<boolean> {
|
||||
|
||||
@ -151,19 +151,7 @@ export async function getEffectiveSettings(): Promise<AppSettings> {
|
||||
...envSettings,
|
||||
};
|
||||
|
||||
// In Basic Auth mode, the "active" search profile is derived from the request
|
||||
// context (basic auth username → profile.basicAuthUser), not from the stored
|
||||
// activeProfileId setting. Expose that derived id so the UI updates the right
|
||||
// profile record on save.
|
||||
const requestOwnerProfileId = getJobOwnerProfileId();
|
||||
if (
|
||||
tenantJobSearchProfile &&
|
||||
requestOwnerProfileId &&
|
||||
requestOwnerProfileId !== DEFAULT_JOB_OWNER_PROFILE_ID &&
|
||||
requestOwnerProfileId !== "__unmapped__"
|
||||
) {
|
||||
result.activeProfileId = requestOwnerProfileId;
|
||||
}
|
||||
|
||||
const rawModel = overrides.model;
|
||||
const modelDef = settingsRegistry.model;
|
||||
@ -254,6 +242,19 @@ export async function getEffectiveSettings(): Promise<AppSettings> {
|
||||
}
|
||||
}
|
||||
|
||||
// In Basic Auth mode, the active search profile is derived from the request
|
||||
// context (username → basicAuthActiveProfileByUser), not the stored
|
||||
// activeProfileId row. Apply after the string-settings loop, which would
|
||||
// otherwise overwrite this field from the global settings table.
|
||||
if (
|
||||
tenantJobSearchProfile &&
|
||||
requestOwnerProfileId &&
|
||||
requestOwnerProfileId !== DEFAULT_JOB_OWNER_PROFILE_ID &&
|
||||
requestOwnerProfileId !== "__unmapped__"
|
||||
) {
|
||||
result.activeProfileId = requestOwnerProfileId;
|
||||
}
|
||||
|
||||
// Always expose the effective base resume id for the active RxResume mode.
|
||||
result.rxresumeBaseResumeId = rxresumeBaseResumeId;
|
||||
|
||||
|
||||
35
package-lock.json
generated
35
package-lock.json
generated
@ -336,6 +336,37 @@
|
||||
"undici-types": "~7.16.0"
|
||||
}
|
||||
},
|
||||
"extractors/google-jobs": {
|
||||
"name": "google-jobs-extractor",
|
||||
"version": "0.0.1",
|
||||
"dependencies": {
|
||||
"camoufox-js": "^0.8.0",
|
||||
"job-ops-shared": "^1.0.0",
|
||||
"playwright": "^1.57.0",
|
||||
"tsx": "^4.4.0"
|
||||
},
|
||||
"devDependencies": {
|
||||
"@types/node": "^24.0.0",
|
||||
"typescript": "~5.9.0"
|
||||
}
|
||||
},
|
||||
"extractors/google-jobs/node_modules/@types/node": {
|
||||
"version": "24.13.3",
|
||||
"resolved": "https://registry.npmjs.org/@types/node/-/node-24.13.3.tgz",
|
||||
"integrity": "sha512-Dh8vAsV36ig5wa9OX4pXvMc9D3Veibfw2wix0CUwYODLD8nkj9UsLjASr49nPg+2eKzxhBV+v7L8pXvT4e639Q==",
|
||||
"dev": true,
|
||||
"license": "MIT",
|
||||
"dependencies": {
|
||||
"undici-types": "~7.18.0"
|
||||
}
|
||||
},
|
||||
"extractors/google-jobs/node_modules/undici-types": {
|
||||
"version": "7.18.2",
|
||||
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.18.2.tgz",
|
||||
"integrity": "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w==",
|
||||
"dev": true,
|
||||
"license": "MIT"
|
||||
},
|
||||
"extractors/gradcracker": {
|
||||
"name": "gradcracker-extractor",
|
||||
"version": "0.0.1",
|
||||
@ -14253,6 +14284,10 @@
|
||||
"url": "https://github.com/sponsors/sindresorhus"
|
||||
}
|
||||
},
|
||||
"node_modules/google-jobs-extractor": {
|
||||
"resolved": "extractors/google-jobs",
|
||||
"link": true
|
||||
},
|
||||
"node_modules/gopd": {
|
||||
"version": "1.2.0",
|
||||
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",
|
||||
|
||||
30
scripts/jobber-cron-dobkin-us.env.example
Normal file
30
scripts/jobber-cron-dobkin-us.env.example
Normal file
@ -0,0 +1,30 @@
|
||||
#
|
||||
# Jobber cron env — U.S. pipeline (optional separate cron host)
|
||||
# Copy to /root/.jobber-cron-dobkin-us.env (chmod 600)
|
||||
#
|
||||
# Note: the U.S. profile is also available in the UI under the same ilia/dobkin
|
||||
# login — use the Jobs page profile picker to switch. This env file is only
|
||||
# needed if you want a dedicated cron that always runs the U.S. profile without
|
||||
# switching in the UI (set BASIC_AUTH_ACTIVE_PROFILE_BY_USER or activate once).
|
||||
#
|
||||
# Used by: scripts/jobber-pipeline-telegram.sh
|
||||
#
|
||||
|
||||
# JobOps base URL (where the app is reachable from the cron host)
|
||||
JOBOPS_URL=http://127.0.0.1:3005
|
||||
|
||||
# Optional: limit number of jobs linked in Telegram message
|
||||
JOB_TELEGRAM_MAX_JOBS=25
|
||||
|
||||
# Optional: comma-separated sources to run (leave empty to use server defaults)
|
||||
# JOBBER_PIPELINE_SOURCES=usajobs,adzuna,remoteok,remotive
|
||||
|
||||
# Same login as Canada; activate the U.S. profile once in Settings/Jobs picker,
|
||||
# or set basicAuthActiveProfileByUser in the DB for this user.
|
||||
BASIC_AUTH_USER=dobkin
|
||||
BASIC_AUTH_PASSWORD=CHANGEME
|
||||
|
||||
# Telegram bot + chat destination
|
||||
TELEGRAM_BOT_TOKEN=CHANGEME
|
||||
TELEGRAM_CHAT_ID=CHANGEME
|
||||
|
||||
@ -340,6 +340,14 @@ const ALL_TARGETS: Target[] = [
|
||||
? undefined
|
||||
: "Camoufox not installed (run: npx camoufox-js fetch)",
|
||||
},
|
||||
{
|
||||
id: "google-jobs",
|
||||
importPath: "../extractors/google-jobs/manifest",
|
||||
settings: { googleJobsMaxJobsPerTerm: "5" },
|
||||
skipReason: camoufoxInstalled()
|
||||
? undefined
|
||||
: "Camoufox not installed (run: npx camoufox-js fetch)",
|
||||
},
|
||||
{
|
||||
id: "workday",
|
||||
importPath: "../extractors/workday/manifest",
|
||||
|
||||
@ -26,6 +26,7 @@ export const EXTRACTOR_SOURCE_IDS = [
|
||||
"testdevjobs",
|
||||
"wellfound",
|
||||
"builtin",
|
||||
"google-jobs",
|
||||
// --- Public ATS / career-page sources ---
|
||||
"ashby",
|
||||
"lever",
|
||||
@ -217,6 +218,12 @@ export const EXTRACTOR_SOURCE_METADATA: Record<
|
||||
category: "pipeline",
|
||||
region: "global",
|
||||
},
|
||||
"google-jobs": {
|
||||
label: "Google Jobs",
|
||||
order: 199,
|
||||
category: "pipeline",
|
||||
region: "global",
|
||||
},
|
||||
ashby: {
|
||||
label: "Ashby (ATS)",
|
||||
order: 210,
|
||||
|
||||
@ -3,8 +3,11 @@ import {
|
||||
buildJobContentFingerprint,
|
||||
buildJobDescriptionFingerprint,
|
||||
collectJobDedupKeys,
|
||||
employersMatchForDismiss,
|
||||
jobsMatchEmployerAndTitleDismiss,
|
||||
normalizeEmployerForFingerprint,
|
||||
normalizeTitleForFingerprint,
|
||||
titlesMatchForDismiss,
|
||||
} from "./job-fingerprint";
|
||||
|
||||
describe("buildJobContentFingerprint", () => {
|
||||
@ -96,10 +99,58 @@ describe("buildJobContentFingerprint", () => {
|
||||
expect(normalizeEmployerForFingerprint("Acme GmbH")).toBe("acme");
|
||||
});
|
||||
|
||||
it("normalizeEmployerForFingerprint strips leading punctuation", () => {
|
||||
expect(normalizeEmployerForFingerprint(". CGI IT UK Limited")).toBe(
|
||||
"cgiituklimited",
|
||||
);
|
||||
});
|
||||
|
||||
it("normalizeTitleForFingerprint drops leading repost markers", () => {
|
||||
expect(normalizeTitleForFingerprint("[Reposted] Software Engineer")).toBe(
|
||||
"softwareengineer",
|
||||
);
|
||||
});
|
||||
});
|
||||
|
||||
describe("prior skip dismiss matching", () => {
|
||||
it("employersMatchForDismiss allows prefix matches", () => {
|
||||
expect(
|
||||
employersMatchForDismiss("cgi", normalizeEmployerForFingerprint("CGI")),
|
||||
).toBe(true);
|
||||
expect(
|
||||
employersMatchForDismiss(
|
||||
normalizeEmployerForFingerprint("CGI IT UK Limited"),
|
||||
normalizeEmployerForFingerprint("CGI"),
|
||||
),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("titlesMatchForDismiss allows plural variants", () => {
|
||||
expect(
|
||||
titlesMatchForDismiss(
|
||||
normalizeTitleForFingerprint("Automation Test Engineer"),
|
||||
normalizeTitleForFingerprint("Automation Test Engineers"),
|
||||
),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("jobsMatchEmployerAndTitleDismiss requires same title family", () => {
|
||||
expect(
|
||||
jobsMatchEmployerAndTitleDismiss({
|
||||
employerA: "CGI IT UK Limited",
|
||||
titleA: "Automation Test Engineers",
|
||||
employerB: "CGI",
|
||||
titleB: "Automation Test Engineer",
|
||||
}),
|
||||
).toBe(true);
|
||||
expect(
|
||||
jobsMatchEmployerAndTitleDismiss({
|
||||
employerA: "Cognizant",
|
||||
titleA: "AI Automation Test Engineer",
|
||||
employerB: "Cognizant",
|
||||
titleB: "Automation Test Engineer",
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
});
|
||||
|
||||
@ -29,6 +29,9 @@ const TRAILING_CITY_REGION_RE = /\s+[-|–—]\s+[^,\-|–—]+,\s*[^,\-|–—]
|
||||
const COMPANY_LEGAL_SUFFIX_RE =
|
||||
/\b(?:inc|inc\.|ltd|ltd\.|llc|gmbh|s\.a\.|s\.r\.l|sa|nv|bv|plc|corp|corporation|co|company|holdings|holding)\b/g;
|
||||
|
||||
/** Hide rediscovered roles that match a prior skip/apply within this window. */
|
||||
export const PRIOR_SKIP_DISMISS_LOOKBACK_MS = 90 * 24 * 60 * 60 * 1000;
|
||||
|
||||
function stripDiacritics(input: string): string {
|
||||
return input.normalize("NFKD").replace(/[\u0300-\u036f]/g, "");
|
||||
}
|
||||
@ -38,6 +41,7 @@ export function normalizeEmployerForFingerprint(
|
||||
): string {
|
||||
if (!employer) return "";
|
||||
let value = stripDiacritics(employer.toLowerCase()).trim();
|
||||
value = value.replace(/^[^a-z0-9]+/g, "");
|
||||
value = value.replace(PARENS_RE, " ");
|
||||
value = value.replace(COMPANY_LEGAL_SUFFIX_RE, " ");
|
||||
value = value.replace(PUNCTUATION_RE, " ");
|
||||
@ -103,6 +107,69 @@ export function buildJobDescriptionFingerprint(args: {
|
||||
return `${employer}::desc::${description}`;
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether two normalized employers refer to the same company for dismiss matching.
|
||||
* Allows prefix matches so `cgi` matches `cgiituklimited`.
|
||||
*/
|
||||
export function employersMatchForDismiss(
|
||||
normalizedEmployerA: string,
|
||||
normalizedEmployerB: string,
|
||||
): boolean {
|
||||
if (!normalizedEmployerA || !normalizedEmployerB) return false;
|
||||
if (normalizedEmployerA === normalizedEmployerB) return true;
|
||||
const [short, long] =
|
||||
normalizedEmployerA.length <= normalizedEmployerB.length
|
||||
? [normalizedEmployerA, normalizedEmployerB]
|
||||
: [normalizedEmployerB, normalizedEmployerA];
|
||||
if (short.length < 3) return false;
|
||||
return long.startsWith(short);
|
||||
}
|
||||
|
||||
/**
|
||||
* Whether two normalized titles are the same role for dismiss matching.
|
||||
* Allows a trailing plural `s` variant (`engineer` vs `engineers`).
|
||||
*/
|
||||
export function titlesMatchForDismiss(
|
||||
normalizedTitleA: string,
|
||||
normalizedTitleB: string,
|
||||
): boolean {
|
||||
if (!normalizedTitleA || !normalizedTitleB) return false;
|
||||
if (normalizedTitleA === normalizedTitleB) return true;
|
||||
if (
|
||||
normalizedTitleA.length > 4 &&
|
||||
normalizedTitleA.endsWith("s") &&
|
||||
normalizedTitleA.slice(0, -1) === normalizedTitleB
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
if (
|
||||
normalizedTitleB.length > 4 &&
|
||||
normalizedTitleB.endsWith("s") &&
|
||||
normalizedTitleB.slice(0, -1) === normalizedTitleA
|
||||
) {
|
||||
return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/** Same company + same job title (normalized), for prior-skip hide rules. */
|
||||
export function jobsMatchEmployerAndTitleDismiss(args: {
|
||||
employerA: string | null | undefined;
|
||||
titleA: string | null | undefined;
|
||||
employerB: string | null | undefined;
|
||||
titleB: string | null | undefined;
|
||||
}): boolean {
|
||||
const employerA = normalizeEmployerForFingerprint(args.employerA);
|
||||
const employerB = normalizeEmployerForFingerprint(args.employerB);
|
||||
const titleA = normalizeTitleForFingerprint(args.titleA);
|
||||
const titleB = normalizeTitleForFingerprint(args.titleB);
|
||||
if (!employerA || !employerB || !titleA || !titleB) return false;
|
||||
return (
|
||||
employersMatchForDismiss(employerA, employerB) &&
|
||||
titlesMatchForDismiss(titleA, titleB)
|
||||
);
|
||||
}
|
||||
|
||||
export function collectJobDedupKeys(args: {
|
||||
employer: string | null | undefined;
|
||||
title: string | null | undefined;
|
||||
|
||||
@ -480,6 +480,13 @@ export const settingsRegistry = {
|
||||
parse: parseIntOrNull,
|
||||
serialize: serializeNullableNumber,
|
||||
},
|
||||
googleJobsMaxJobsPerTerm: {
|
||||
kind: "typed" as const,
|
||||
schema: z.number().int().min(1).max(150),
|
||||
default: (): number => 30,
|
||||
parse: parseIntOrNull,
|
||||
serialize: serializeNullableNumber,
|
||||
},
|
||||
fourdayweekMaxJobsPerTerm: {
|
||||
kind: "typed" as const,
|
||||
schema: z.number().int().min(1).max(1000),
|
||||
@ -965,6 +972,10 @@ export const settingsRegistry = {
|
||||
kind: "string" as const,
|
||||
schema: z.string().trim().max(200),
|
||||
},
|
||||
basicAuthActiveProfileByUser: {
|
||||
kind: "string" as const,
|
||||
schema: z.string().trim().max(10000),
|
||||
},
|
||||
rxresumeBaseResumeId: {
|
||||
kind: "string" as const,
|
||||
schema: z.string().trim().max(200),
|
||||
|
||||
@ -182,6 +182,7 @@ export const createAppSettings = (
|
||||
override: null,
|
||||
},
|
||||
activeProfileId: null,
|
||||
basicAuthActiveProfileByUser: null,
|
||||
rxresumeBaseResumeId: null,
|
||||
rxresumeBaseResumeIdV4: null,
|
||||
rxresumeBaseResumeIdV5: null,
|
||||
@ -206,6 +207,7 @@ export const createAppSettings = (
|
||||
builtinMaxJobsPerTerm: { value: 100, default: 100, override: null },
|
||||
builtinMaxPagesPerTerm: { value: 3, default: 3, override: null },
|
||||
wellfoundMaxJobsPerTerm: { value: 50, default: 50, override: null },
|
||||
googleJobsMaxJobsPerTerm: { value: 30, default: 30, override: null },
|
||||
fourdayweekMaxJobsPerTerm: { value: 50, default: 50, override: null },
|
||||
qajobsboardMaxJobsPerTerm: { value: 100, default: 100, override: null },
|
||||
arcRemoteJobsPaths: {
|
||||
|
||||
@ -218,6 +218,7 @@ export interface AppSettings {
|
||||
builtinMaxJobsPerTerm: Resolved<number>;
|
||||
builtinMaxPagesPerTerm: Resolved<number>;
|
||||
wellfoundMaxJobsPerTerm: Resolved<number>;
|
||||
googleJobsMaxJobsPerTerm: Resolved<number>;
|
||||
fourdayweekMaxJobsPerTerm: Resolved<number>;
|
||||
qajobsboardMaxJobsPerTerm: Resolved<number>;
|
||||
arcRemoteJobsPaths: Resolved<string[]>;
|
||||
@ -270,6 +271,8 @@ export interface AppSettings {
|
||||
|
||||
// Simple strings:
|
||||
activeProfileId: string | null;
|
||||
/** JSON map of basic-auth username → search profile id when one login has multiple profiles. */
|
||||
basicAuthActiveProfileByUser: string | null;
|
||||
rxresumeBaseResumeId: string | null;
|
||||
rxresumeBaseResumeIdV4: string | null;
|
||||
rxresumeBaseResumeIdV5: string | null;
|
||||
|
||||
Loading…
x
Reference in New Issue
Block a user