feat: add Google Jobs extractor and fix scraper reliability gaps
All checks were successful
CI / skip-ci-check (pull_request) Successful in 8s
CI / docker-ci (pull_request) Successful in 10s
CI / secret-scan (pull_request) Successful in 7s

Ship a Camoufox-backed Google Jobs source, restore Glassdoor results discarded by a python-jobspy GraphQL quirk, and fix Himalayas/Gradcracker zero-job failures. Also harden prior-skip dismiss matching and multi-profile basic-auth switching for CA/US runs.
This commit is contained in:
ilia 2026-07-09 15:11:53 -04:00
parent 84e6835b11
commit 40c7cdece3
47 changed files with 1989 additions and 87 deletions

View File

@ -260,7 +260,35 @@ ADZUNA_APP_KEY=
# =============================================================================
# Caps via Settings: workingnomadsMaxJobsPerTerm, testdevjobsMaxJobsPerTerm,
# testdevjobsMaxPages, builtinMaxJobsPerTerm, builtinMaxPagesPerTerm,
# wellfoundMaxJobsPerTerm (Wellfound uses Camoufox when installed).
# wellfoundMaxJobsPerTerm, googleJobsMaxJobsPerTerm (Wellfound and Google
# Jobs both use Camoufox when installed).
# =============================================================================
# Google Jobs — optional (auto-enabled, no credentials, unofficial scrape)
# =============================================================================
# GOOGLE_JOBS_PROXY=http://user:pass@proxy-host:port
# ^ Most reliable way to avoid Google's CAPTCHA/rate-limit wall (residential or
# mobile proxy recommended; datacenter IPs, including most VPS/cloud hosts,
# are aggressively blocked). Applies to both manual runs and the pipeline.
#
# GOOGLE_JOBS_HEADLESS=false
# ^ Manual/local use ONLY (do not set this in the server .env for the
# unattended pipeline/cron — a headless server has no display for a human
# to see the browser). Opens a real Firefox window so you can solve a
# CAPTCHA by hand; waits up to GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS (default
# 300000 = 5 min) before giving up. Run directly, e.g.:
# GOOGLE_JOBS_HEADLESS=false npm --workspace google-jobs-extractor run start
# Session cookies are saved to extractors/google-jobs/storage/state.json
# and reused by later runs (including headless/automated ones), which can
# reduce how often future runs hit a CAPTCHA.
#
# Caps via Settings: googleJobsMaxJobsPerTerm (default 30, max 150).
#
# Note: there is no country/location override for this extractor — Google
# resolves location from the browser's real (or GOOGLE_JOBS_PROXY's) IP.
# Forcing a mismatched country has been observed to make Google's Jobs
# vertical return "no matches" even though the tab loads correctly. To
# target a different country's jobs, use a proxy with an exit node there.
# =============================================================================
# Teamtailor / Huntflow / Factorial / Career pages — optional

View File

@ -12,8 +12,9 @@
"!!**/.venv",
"!!docs-site/.docusaurus",
"!!docs-site/build",
"!!extractors/jobspy/storage",
"!!extractors/*/storage",
"!!orchestrator/storage",
"!!storage",
"!!data"
]
},

View File

@ -32,10 +32,11 @@ Eluta surfaces Canadian roles indexed directly from employer career sites, often
### Empty feeds
- The `location` string may be too broad or spelled differently than Eluta expects. Try a major city plus province (e.g. `Calgary, AB`).
- Eluta's RSS is a small recent slice of *all* jobs for that location (often retail/hospitality-heavy). Pipeline search terms like `software engineer` can legitimately match zero items even when the feed itself is healthy. Broaden terms or add more metro locations.
### RSS HTTP errors
### RSS HTTP errors / "Too Many Requests"
- Eluta may block unusual clients; the extractor sends a conventional User-Agent. Retry later or reduce the number of location feeds per run.
- Eluta rate-limits aggressive or browser-like clients. The extractor uses a dedicated `JobOps/... Eluta RSS consumer` User-Agent; retry later or reduce the number of location feeds per run.
## Related pages

View File

@ -0,0 +1,106 @@
---
id: google-jobs
title: Google Jobs Extractor
description: Browser-backed scraping of the Google for Jobs search widget.
sidebar_position: 8
---
## What it is
Original feature: [Google for Jobs](https://www.google.com/search?q=jobs&udm=8)
Google Jobs is a browser-backed extractor that opens Google's Jobs vertical (`udm=8`) in Firefox (Camoufox when available) for each pipeline search term, scrolls the results panel to load additional cards, then clicks into each card to resolve a real "Apply on <site>" link and description before mapping rows into the orchestrator `CreateJobInput` shape.
Implementation split:
1. `extractors/google-jobs/src/main.ts` builds the search URL, waits for the jobs list, scrolls to lazy-load more cards, extracts card metadata (title/employer/location/"via" source), clicks each card to read the detail pane, and writes dataset JSON.
2. `extractors/google-jobs/src/run.ts` spawns the browser subprocess, applies a hard timeout, reads the dataset, and maps rows to `CreateJobInput`.
3. `extractors/google-jobs/manifest.ts` wires pipeline `searchTerms` into the extractor.
### A note on the search URL and location
Google's Jobs vertical is reached with `udm=8` (its current internal vertical id). An older, commonly-documented deep link — `ibp=htl;jobs` — still works but gets silently redirected to `udm=8` anyway, and combining it with extra parameters (a `num=` result-count hint, or a `gl=` country override that doesn't match the browser's real/proxied IP) has been observed to make Google return "no matches" or "can't determine location" even though the tab itself loads correctly.
Because of that, this extractor does **not** send a `gl` (country) parameter — location is resolved by Google from the browser's real IP (or `GOOGLE_JOBS_PROXY`'s exit IP), which Camoufox's `geoip` option keeps internally consistent. To target a different country's jobs, use a residential proxy with an exit node in that country rather than trying to override the country via a parameter.
### Dealing with Google's CAPTCHA / bot detection
Google actively fingerprints and challenges automated browser traffic, especially from datacenter/VPS IPs (which most servers running JobOps' cron pipeline will have). The extractor has layered mitigations, from always-on to opt-in:
1. **Always on:** Camoufox launches with `humanize: true` and `geoip: true` (human-like cursor movement, IP-consistent locale/geolocation) to reduce — not eliminate — fingerprint-based detection.
2. **Always on:** browser cookies/session are persisted to `extractors/google-jobs/storage/state.json` and reused on the next run. A session that has cleared a CAPTCHA once (including via option 4 below) tends to see fewer challenges for a while afterward.
3. **Opt-in, most reliable:** set `GOOGLE_JOBS_PROXY` (e.g. a residential/mobile proxy URL) to avoid datacenter-IP blocks in the first place. This is the standard real-world fix; unproxied runs from cloud servers should expect to be blocked often.
4. **Opt-in, manual/local only:** set `GOOGLE_JOBS_HEADLESS=false` and run the extractor directly (not via the pipeline) to open a real, visible Firefox window. If Google shows a CAPTCHA, solve it yourself in that window — the script polls and automatically resumes once it clears (up to `GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS`, default 5 minutes). The resulting session cookies are saved and reused by later automated/headless runs. **Do not** set `GOOGLE_JOBS_HEADLESS=false` in the server's always-on `.env` — a headless server has no display for a human to interact with, and the run will just wait until the timeout.
## Why it exists
Google for Jobs aggregates postings across the open web — including boards and employer career pages that JobOps does not otherwise scrape. Adding it as a proper source closes some of the gap between "jobs found by manually Googling a role" and "jobs discovered by the pipeline."
It is **not** a general web crawler and does not replace dedicated extractors: it only surfaces whatever Google's Jobs widget chooses to show for a query, subject to Google's own ranking, freshness, and regional availability.
## How to use it
1. Open **Run jobs** and choose **Automatic**.
2. **Google Jobs** is enabled by default in **Sources** (toggle it off if you do not want it for this run).
3. Set your existing automatic run knobs:
- `searchTerms` become the `"<term> jobs"` query sent to Google per term.
- `googleJobsMaxJobsPerTerm` (Settings) caps jobs scraped per run (default `30`, max `150`).
- Location is derived from the browser's real/proxied IP, not from a country setting — see [the note above](#a-note-on-the-search-url-and-location).
4. Start the run and watch progress in the pipeline progress card.
Local run example (headless, no proxy):
```bash
GOOGLE_JOBS_SEARCH_TERMS='["automation engineer"]' \
GOOGLE_JOBS_MAX_JOBS='10' \
npm --workspace google-jobs-extractor run start
```
Local run example (headed, solve the CAPTCHA yourself if one appears):
```bash
GOOGLE_JOBS_HEADLESS=false \
GOOGLE_JOBS_SEARCH_TERMS='["automation engineer"]' \
npm --workspace google-jobs-extractor run start
```
Defaults and constraints:
- No credentials required; this is a direct HTML/DOM scrape of Google's public search results, not an official API (Google does not offer a free Google Jobs API).
- Google's markup and CSS class names change frequently and are obfuscated; the scraper favors known current card classes (e.g. `.EimVGf`) with multiple fallback selectors and a leaf-node text heuristic, but selectors can still rot and return zero results until updated.
- Every card carries Google's own `data-share-url` permalink for that specific listing, which the extractor uses as `jobUrl` — so a job stays unique and reviewable even if the "click into the card for a detail pane" step below fails for it.
- Because resolving a real application link requires clicking each card in a real browser, this extractor is slower per job than API-backed sources — keep `googleJobsMaxJobsPerTerm` modest for frequent runs.
- If no external "Apply on &lt;site&gt;" link is found for a card, the extractor falls back to Google's own permalink for that listing so it's still reviewable, but not directly one-click-applyable.
- Automated, frequent, or high-volume scraping of Google search results may be against Google's Terms of Service; treat this extractor as best-effort and keep run frequency/volume conservative.
- See [Dealing with Google's CAPTCHA / bot detection](#dealing-with-googles-captcha--bot-detection) above for proxy and interactive-solve options.
## Common problems
### Google Jobs returns 0 jobs for a term
The extractor logs one of two distinct messages to help tell these apart:
- `Google's Jobs tab reported no matches for "<term>"` — Google correctly landed on the Jobs tab but genuinely has no results for that query/locale. Try a broader, more common phrasing (e.g. `"automation engineer"` instead of a very narrow title).
- `Jobs list did not appear for "<term>"` — an unrecognized page state (consent wall, layout change, or rotted selectors). The extractor saves a screenshot and full HTML dump to `extractors/google-jobs/storage/debug-last.png` / `debug-last.html` for inspection; compare it against a fresh manual search to see what changed.
### Results look sparse or missing company/location
- The list-view heuristics rely on leaf DOM nodes near the job title; if Google restructures the card markup, company/location extraction can degrade even when titles still work. This is expected best-effort behavior for an unofficial scrape.
### Run is slow
- Each job requires a real click + detail-pane render, unlike single-request API extractors. Lower `googleJobsMaxJobsPerTerm` or reduce the number of search terms for faster runs.
### Blocked, CAPTCHA'd, or consistently empty results
- Google may rate-limit or challenge automated browser traffic, especially from datacenter/VPS IPs. In order of effort:
1. Set `GOOGLE_JOBS_PROXY` to a residential/mobile proxy — the most reliable fix for unattended/server runs.
2. Run locally with `GOOGLE_JOBS_HEADLESS=false` and solve the CAPTCHA yourself once; the saved session (`storage/state.json`) may reduce blocks on subsequent headless runs for a while.
3. Reduce run frequency, lower the per-term cap, or disable the source for that run and rely on other extractors / Manual Import instead.
## Related pages
- [Extractors Overview](/docs/next/extractors/overview)
- [Manual Import](/docs/next/extractors/manual)
- [Pipeline Run](/docs/next/features/pipeline-run)
- [Settings](/docs/next/features/settings)

View File

@ -46,3 +46,11 @@ Set `JOBOPS_EMIT_PROGRESS=1` for structured progress lines consumable by orchest
- Uses Playwright + Crawlee via Camoufox.
- Low concurrency and longer timeouts for stability.
- Employer logo (`figure img` alt text) and employer link are optional. Cards that omit them used to throw a Playwright `getAttribute` timeout and fail the whole region page (0 jobs saved even after cards were counted). Missing employer fields are now skipped per card.
## Common problems
### Region pages time out / run returns 0 jobs
- Gradcracker Livewire markup sometimes omits the employer logo on a card. Older builds aborted the whole page on that timeout. Current builds continue and leave employer blank.
- Anti-bot / Camoufox issues: ensure Camoufox is installed (`npx camoufox-js fetch`) and prefer headed runs when debugging.

View File

@ -0,0 +1,45 @@
---
id: himalayas
title: Himalayas Extractor
description: Public remote-jobs API pagination and client-side term filtering.
sidebar_position: 16
---
## What it is
Original site: [himalayas.app](https://himalayas.app)
The extractor lives in `extractors/himalayas/manifest.ts`. It paginates the public JSON API (`https://himalayas.app/jobs/api?limit=&offset=`), filters rows client-side by pipeline search terms (title + categories), and maps matches into `CreateJobInput`.
## Why it exists
Himalayas is a large remote-jobs index with a stable, unauthenticated API — useful for remote-first discovery without browser automation or credentials.
## How to use it
1. Enable **Himalayas** in pipeline sources.
2. Optionally set **Himalayas max jobs per term** (`himalayasMaxJobsPerTerm`, default `100`).
3. Pipeline `searchTerms` filter titles/categories client-side (substring match, case-insensitive).
4. Run the pipeline.
### Defaults and constraints
- The upstream API **silently caps `limit` at 20** per request. The extractor uses `PAGE_SIZE = 20` and paginates with `offset` (up to 10 pages / 200 rows scanned per run before the per-term cap).
- There is no server-side search — if a term is rare, early pages may not contain matches even though later pages do. Raise the page budget only by changing the extractor constants if you need deeper scans.
- No auth required.
## Common problems
### Zero jobs for a term I know exists
- The term may not appear in the first ~200 API rows (newest-first feed). Try a shorter token (`software`, `engineer`, `SDET`) or accept that Himalayas skews toward whatever is currently at the top of the feed.
- Confirm you are not hitting an old build that requested `limit=50` and stopped after one page (fixed: the API returns at most 20, which used to look like end-of-feed).
### Rate limits / HTTP errors
- Retry later; the public API is generally stable but can throttle burst traffic.
## Related pages
- [Extractors Overview](/docs/next/extractors/overview)
- [Add an Extractor](/docs/next/workflows/add-an-extractor)

View File

@ -61,4 +61,8 @@ The service in `orchestrator/src/server/services/jobspy.ts`:
- A run returned fewer LinkedIn descriptions than expected.
`JOBSPY_LINKEDIN_FETCH_DESCRIPTION=0` disables description fetching to speed up runs.
- Different cities need different workplace-type filters.
This is not supported in the current automatic-run flow. JobSpy receives one global workplace-type selection per run/query invocation.
This is not supported in the current automatic-run flow. JobSpy receives one global workplace-type selection per run/query invocation.
- Glassdoor always returns 0 jobs (fixed).
Upstream `python-jobspy` (through at least `1.1.82`) discards an entire page of Glassdoor results whenever Glassdoor's GraphQL response includes *any* `errors` entry — even for an unrelated field. In practice, Glassdoor's backend consistently returns a partial error for `jobsPageSeoData` (`SUBREQUEST_HTTP_ERROR` / "dns error" on Glassdoor's own infra) while the actual job listings still come back fine, so every Glassdoor run silently returned 0 jobs. `extractors/jobspy/scrape_jobs.py` monkeypatches `Glassdoor._fetch_jobs_page` at import time to only treat the response as failed when `data.jobListings` is actually missing, and logs a warning (not an error) when it ignores an unrelated partial error. `requirements.txt` pins `python-jobspy==1.1.82` so an upstream version bump doesn't silently drop this patch — re-verify the patched method still matches before removing the pin.
- A Glassdoor run logs `429` / "Blocked by Glassdoor for too many requests".
This is a real rate limit from running too many Glassdoor requests back-to-back (e.g. repeated manual test runs). Space out runs; the automatic pipeline's normal cadence does not trigger this in practice.

View File

@ -25,6 +25,7 @@ Extractor integrations are now registered through manifests and loaded automatic
| [Eluta](/docs/next/extractors/eluta) | Canadian listings aggregated from employer career sites (RSS) | Canada-only source (skipped when search geography is not Canada); RSS `location` strings must be set | `ELUTA_RSS_LOCATIONS`, `ELUTA_MAX_JOBS_PER_TERM` | Fetches one or more `eluta.ca` RSS feeds, filters by terms, de-duplicates by guid/URL |
| [QAJobsBoard](/docs/next/extractors/qajobsboard) | QA / SDET / automation-heavy board (global JSON feed) | No auth; geography skew is manual/filter downstream | `qajobsboardMaxJobsPerTerm` | Fetches JobBoardly JSON, filters by pipeline terms |
| [Arc.dev](/docs/next/extractors/arcdev) | Remote roles from Arc.dev listing pages (tool-tagged paths) | Parses SSR `__NEXT_DATA__`; relies on stable Next payload | `ARC_REMOTE_JOBS_PATHS` (seeds defaults), `arcRemoteJobsPaths`, `arcMaxJobsPerPath` | Merges Arc-managed + external rows; dedupes by URL |
| [Google Jobs](/docs/next/extractors/google-jobs) | Broader open-web coverage via the Google for Jobs widget | Unofficial scrape (no API); CAPTCHA-prone from datacenter IPs; selectors can rot; slower (click-through per job) | `googleJobsMaxJobsPerTerm`, `GOOGLE_JOBS_PROXY`, `GOOGLE_JOBS_HEADLESS` | Browser scrape of `udm=8` Jobs vertical; scrolls list, double-clicks cards, expands description |
| [Manual Import](/docs/next/extractors/manual) | One-off jobs not covered by scrapers | Inference quality depends on model/provider and input quality; some URLs cannot be fetched reliably | App/API endpoints (`/api/manual-jobs/infer`, `/api/manual-jobs/import`) | Accepts text/HTML/URL, runs inference, then saves and scores job after review |
## Which extractor should I use?
@ -40,6 +41,7 @@ Extractor integrations are now registered through manifests and loaded automatic
- Use **BC T-Net** for British Columbia tech RSS listings (runs only when search geography is Canada).
- Use **Eluta** for Canadian employer-direct listings via RSS (set metro/province `location` strings).
- Use **QAJobsBoard** or **Arc.dev** when you want QA- or remote-stack-focused feeds without extra credentials.
- Use **Google Jobs** when you want broader open-web coverage beyond dedicated job boards (unofficial scrape; best-effort).
- Use **Manual Import** when you already have a specific posting and need direct import.
Many runs combine sources: broad discovery first, then manual import for high-priority jobs that scraping misses.
@ -82,5 +84,7 @@ JobOps ships **BC T-Net** and **iCIMS tenant HTML** extractors for two cases tha
- [Arc.dev](/docs/next/extractors/arcdev)
- [Canadian / NA QA contracting firms](/docs/next/extractors/qa-contract-staffing-canada)
- [Canadian companies — QA-strong ATS](/docs/next/extractors/canadian-companies-qa-ats)
- [Google Jobs](/docs/next/extractors/google-jobs)
- [Himalayas](/docs/next/extractors/himalayas)
- [Manual Import](/docs/next/extractors/manual)
- [Add an Extractor](/docs/next/workflows/add-an-extractor)

View File

@ -45,9 +45,19 @@ During a pipeline run, if a new posting matches an existing row by URL, source i
Open jobs that match a prior skip or apply are **hidden** from Discovered, Ready, and All tabs so the queue stays fresh. Skipped and applied rows themselves remain visible in their statuses.
Use **Filters → Employer keywords → Hide roles you already skipped** to toggle this behavior (on by default). When enabled, JobOps compares **company + job title** only:
- Same company after normalization (legal suffixes stripped; short names like `CGI` match longer forms like `CGI IT UK Limited`).
- Same title after normalization (including simple plural variants such as `Engineer` vs `Engineers`).
- Prior skip or apply must be within the **last 90 days** (`updatedAt` on the skipped/applied row).
Uncheck the filter to temporarily review rediscovered rows that match older skips.
## Defaults and constraints
- Description matching requires at least **80 characters** of normalized text; short or empty descriptions fall back to employer+title only.
- Prior-skip hiding uses employer + title only (not description text).
- The 90-day window applies to both the Jobs list filter and pipeline import suppression for skipped/applied rows.
- Description matching during import still requires at least **80 characters** of normalized text; short or empty descriptions fall back to employer+title only.
- Matching is **per profile** (`ownerProfileId`); different login profiles do not share dedup state.
- Dedup does **not** delete existing rows retroactively when you change skip list or country filters — run discovery again or skip manually for old data.
- Very different titles at the same company (for example `SDET` vs `Product Designer`) are **not** collapsed.

View File

@ -0,0 +1,49 @@
import type {
ExtractorManifest,
ExtractorRunResult,
} from "@shared/types/extractors";
import { runGoogleJobs } from "./src/run.js";
export const manifest: ExtractorManifest = {
id: "google-jobs",
displayName: "Google Jobs",
providesSources: ["google-jobs"],
async run(context): Promise<ExtractorRunResult> {
if (context.shouldCancel?.()) return { success: true, jobs: [] };
const parsedMax = context.settings.googleJobsMaxJobsPerTerm
? Number.parseInt(context.settings.googleJobsMaxJobsPerTerm, 10)
: Number.NaN;
const maxJobs = Number.isFinite(parsedMax) ? Math.max(1, parsedMax) : 30;
context.onProgress?.({
phase: "list",
termsProcessed: 0,
termsTotal: context.searchTerms.length || 1,
currentUrl: "https://www.google.com/search?udm=8",
detail: "Google Jobs: launching browser scrape",
});
const result = await runGoogleJobs({
searchTerms: context.searchTerms,
maxJobs,
});
if (!result.success) {
return { success: false, jobs: [], error: result.error };
}
context.onProgress?.({
phase: "list",
termsProcessed: context.searchTerms.length || 1,
termsTotal: context.searchTerms.length || 1,
currentUrl: "https://www.google.com/search?udm=8",
jobPagesProcessed: result.jobs.length,
detail: `Google Jobs: ${result.jobs.length} jobs`,
});
return { success: true, jobs: result.jobs };
},
};
export default manifest;

View File

@ -0,0 +1,22 @@
{
"name": "google-jobs-extractor",
"version": "0.0.1",
"type": "module",
"description": "Google Jobs (Google for Jobs) search widget extractor (browser-backed)",
"main": "manifest.ts",
"dependencies": {
"camoufox-js": "^0.8.0",
"job-ops-shared": "^1.0.0",
"playwright": "^1.57.0",
"tsx": "^4.4.0"
},
"devDependencies": {
"@types/node": "^24.0.0",
"typescript": "~5.9.0"
},
"scripts": {
"start": "tsx src/main.ts",
"check:types": "tsc --noEmit",
"get-binaries": "camoufox-js fetch"
}
}

View File

@ -0,0 +1,616 @@
import { access, mkdir, writeFile } from "node:fs/promises";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { launchOptions } from "camoufox-js";
import {
type Browser,
type BrowserContext,
firefox,
type Page,
} from "playwright";
const __dirname = dirname(fileURLToPath(import.meta.url));
const OUTPUT_PATH = join(__dirname, "../storage/jobs.json");
const STATE_PATH = join(__dirname, "../storage/state.json");
const DEBUG_HTML_PATH = join(__dirname, "../storage/debug-last.html");
const DEBUG_SCREENSHOT_PATH = join(__dirname, "../storage/debug-last.png");
interface ScrapedJob {
title: string;
employer: string;
jobUrl: string;
applicationLink?: string;
location?: string;
jobDescription?: string;
postedVia?: string;
isRemote?: boolean;
}
interface ScrapedCard {
docId?: string;
title: string;
employer: string;
location?: string;
via?: string;
/** Google's own permalink into this specific listing (from `data-share-url`). Always present when the card matched, so it works as a stable jobUrl even if the detail click fails. */
shareUrl?: string;
}
interface ScrapedDetail {
applyLinks: Array<{ href: string; text: string }>;
description?: string;
}
/**
* Confirmed against Google's live markup (2026). Google renders each job as
* a `.EimVGf` card with a `data-share-url` permalink baked in no `<ul>`/
* `<li>`/`aria-label="Jobs list"` wrapper as older scraping guides assumed.
* Kept as a short list (not a single hardcoded class) since Google rotates
* these obfuscated names periodically.
*/
const CARD_SELECTORS = [".EimVGf", 'li[role="listitem"]', ".iFjolb"] as const;
function parseTerms(raw: string | undefined): string[] {
if (!raw) return ["software engineer"];
try {
const parsed = JSON.parse(raw) as unknown;
if (Array.isArray(parsed)) {
return parsed.map((entry) => String(entry).trim()).filter(Boolean);
}
} catch {
// fall through to delimited parsing
}
return raw
.split(/[\n|,]+/)
.map((entry) => entry.trim())
.filter(Boolean);
}
/**
* `udm=8` is Google's current internal id for the Jobs vertical (the legacy
* `ibp=htl;jobs` deep link now gets silently redirected to it anyway, but
* only when requested directly combining the two, or adding `num=`/`gl=`
* overrides that don't match the browser's real/proxied IP, has been
* observed to make the vertical come back "no matches" or "can't determine
* location" even though the widget itself loads correctly). `hl=en` keeps
* copy in English so downstream text parsing (e.g. "via LinkedIn") is
* consistent regardless of where the request is geolocated. Location is
* intentionally NOT set via a `gl` param it's derived from the real (or
* `GOOGLE_JOBS_PROXY`) IP through Camoufox's `geoip` option instead.
*/
function buildSearchUrl(term: string): string {
const params = new URLSearchParams({
q: `${term} jobs`,
udm: "8",
hl: "en",
});
return `https://www.google.com/search?${params.toString()}`;
}
/**
* Launches Camoufox-hardened Firefox. `humanize`/`geoip` reduce (but do not
* eliminate) Google's automated-traffic detection. `GOOGLE_JOBS_PROXY` (e.g.
* a residential proxy URL) is the most reliable way to avoid datacenter-IP
* blocks entirely, if configured.
*/
async function launchBrowser(headless: boolean): Promise<Browser> {
const proxy = process.env.GOOGLE_JOBS_PROXY?.trim();
return firefox.launch(
await launchOptions({
headless,
humanize: true,
geoip: true,
...(proxy ? { proxy } : {}),
}),
);
}
/** Google shows a cookie/consent wall in some locales before rendering results. */
async function dismissConsent(page: Page): Promise<void> {
const labels = ["Reject all", "I agree", "Accept all"];
for (const label of labels) {
try {
const button = page.getByRole("button", { name: label }).first();
if (await button.isVisible({ timeout: 1500 })) {
await button.click({ timeout: 3000 });
await page.waitForTimeout(500);
return;
}
} catch {
// selector not present in this render; try the next label
}
}
}
/** Thrown when Google serves a CAPTCHA / rate-limit challenge instead of results. */
class GoogleBlockedError extends Error {}
/** Google serves an interstitial CAPTCHA page for automated / datacenter traffic. */
async function isBlockedPage(page: Page): Promise<boolean> {
if (/\/sorry\//.test(page.url())) return true;
try {
return await page.evaluate(() => {
const bodyText = document.body?.textContent ?? "";
return (
document.querySelector("#captcha-form") !== null ||
document.querySelector(".g-recaptcha") !== null ||
/unusual traffic/i.test(bodyText)
);
});
} catch {
return false;
}
}
/**
* In headed/interactive mode, give a human at the keyboard a chance to solve
* the CAPTCHA in the visible browser window instead of failing immediately.
* Polls until the challenge clears (Google's form redirects on success) or
* the timeout elapses.
*/
async function waitForHumanToUnblock(
page: Page,
timeoutMs: number,
): Promise<boolean> {
const deadline = Date.now() + timeoutMs;
console.error(
`[google-jobs] Google is showing a CAPTCHA in the open browser window — please solve it there. Waiting up to ${Math.round(timeoutMs / 1000)}s...`,
);
while (Date.now() < deadline) {
await page.waitForTimeout(3_000);
if (!(await isBlockedPage(page))) {
console.error("[google-jobs] CAPTCHA cleared, resuming.");
return true;
}
}
return false;
}
async function waitForJobsList(page: Page): Promise<boolean> {
for (const selector of CARD_SELECTORS) {
try {
await page.waitForSelector(selector, { timeout: 8_000 });
return true;
} catch {
// try the next fallback selector
}
}
return false;
}
/**
* Google sometimes lands correctly on the Jobs vertical/tab but reports no
* matches for the query (distinct from a CAPTCHA block or a selector-rot
* failure) e.g. "It looks like there aren't any 'Jobs' matches on this topic."
*/
async function isEmptyJobsVertical(page: Page): Promise<boolean> {
try {
return await page.evaluate(() => {
const bodyText = document.body?.textContent ?? "";
return /aren.t any .Jobs. matches/i.test(bodyText);
});
} catch {
return false;
}
}
function countCardsScript(selectors: readonly string[]): number {
for (const selector of selectors) {
const count = document.querySelectorAll(selector).length;
if (count > 0) return count;
}
return 0;
}
/** Google lazy-loads additional cards (via a custom `<infinity-scrolling>` element) as the page is scrolled. */
function scrollPageScript(): number {
const before = window.scrollY;
window.scrollTo(0, document.body.scrollHeight);
return window.scrollY - before;
}
async function growJobsList(
page: Page,
selectors: readonly string[],
targetCount: number,
): Promise<void> {
for (let attempt = 0; attempt < 8; attempt += 1) {
const count = await page.evaluate(countCardsScript, selectors);
if (count >= targetCount) return;
const scrolled = await page.evaluate(scrollPageScript);
if (scrolled <= 0) return;
await page.waitForTimeout(900);
}
}
function extractCardsScript(selectors: readonly string[]): ScrapedCard[] {
// NOTE: this whole function is serialized via Function.prototype.toString()
// and re-run inside the browser page by Playwright — it cannot reference
// anything from the outer module scope (constants, other functions, etc.).
// All helpers must be declared inside this function body.
function clean(value: string | null | undefined): string {
return (value ?? "").replace(/\s+/g, " ").trim();
}
/** Extracts Google's internal `htidocid` from a job's `data-share-url`, matching `data-encoded-docid` in the detail panel's apply-link list. */
function extractDocId(shareUrl: string | null): string | undefined {
if (!shareUrl) return undefined;
try {
const url = new URL(shareUrl, "https://www.google.com");
return url.searchParams.get("htidocid") ?? undefined;
} catch {
return undefined;
}
}
function findCards(): Element[] {
for (const selector of selectors) {
const items = Array.from(document.querySelectorAll(selector));
if (items.length > 0) return items;
}
return [];
}
const cards: ScrapedCard[] = [];
for (const item of findCards()) {
// Known current classes first (title/employer/location+via), falling
// back to a leaf-node text heuristic if Google has rotated them.
const titleEl = item.querySelector(".tNxQIb, h2, [role='heading']");
const title = clean(titleEl?.textContent);
if (!title) continue;
const wHYlTdEls = Array.from(item.querySelectorAll(".wHYlTd"));
let employer = clean(wHYlTdEls[0]?.textContent);
let location: string | undefined;
let via: string | undefined;
const viaBlock = wHYlTdEls.find((el) =>
/\bvia\b/i.test(el.textContent ?? ""),
);
if (viaBlock) {
const text = clean(viaBlock.textContent);
const match = text.match(/^(.*?)\s*(?:•|·)\s*via\s+(.+)$/i);
if (match) {
location = match[1] || undefined;
via = match[2] || undefined;
} else {
via = clean(text.replace(/^via\s*/i, ""));
}
}
if (!employer || !title || employer === title) {
// Selector rot fallback: scan leaf text nodes structurally.
const leafTexts = Array.from(item.querySelectorAll("div, span"))
.filter((el) => el.children.length === 0)
.map((el) => clean(el.textContent))
.filter((text) => text && text !== title && !/^[•·|,]+$/.test(text));
employer = employer || leafTexts[0] || "Unknown Employer";
if (!via) {
const viaText = leafTexts.find((text) => /^via\b/i.test(text));
via = viaText ? clean(viaText.replace(/^via\s*/i, "")) : undefined;
location =
location ?? leafTexts.slice(1).find((text) => text !== viaText);
}
}
const shareUrlRaw = item.getAttribute("data-share-url") ?? undefined;
const docId = extractDocId(shareUrlRaw ?? null);
cards.push({
docId,
title,
employer: employer || "Unknown Employer",
location,
via,
shareUrl: shareUrlRaw,
});
}
return cards;
}
/**
* Scopes apply-link extraction to the clicked job's own `data-encoded-docid`
* block. Google's detail viewer can have adjacent/related jobs' apply links
* present in the DOM simultaneously, so a document-wide "first apply link"
* scan can silently pick up the wrong job's link.
*/
function extractDetailScript(docId: string | undefined): ScrapedDetail {
function clean(value: string | null | undefined): string {
return (value ?? "").replace(/\s+/g, " ").trim();
}
function collectApplyLinks(
root: ParentNode,
): Array<{ href: string; text: string }> {
const links: Array<{ href: string; text: string }> = [];
for (const anchor of Array.from(root.querySelectorAll("a[href]"))) {
const href = (anchor as HTMLAnchorElement).href;
const text =
clean(anchor.textContent) || clean(anchor.getAttribute("aria-label"));
if (!href) continue;
if (
!/apply/i.test(text) &&
!/apply/i.test(anchor.getAttribute("aria-label") ?? "")
) {
continue;
}
if (
href.includes("google.com/search") ||
href.includes("accounts.google.com")
) {
continue;
}
links.push({ href, text });
}
return links;
}
let applyLinks: Array<{ href: string; text: string }> = [];
if (docId) {
const scoped = document.querySelector(`[data-encoded-docid="${docId}"]`);
if (scoped) applyLinks = collectApplyLinks(scoped);
}
if (applyLinks.length === 0) {
applyLinks = collectApplyLinks(document);
}
// `.textContent` also pulls in text from nested <style>/<script> tags
// (Google embeds scoped theme <style> blocks inside content divs) and from
// the job-list sidebar itself (which concatenates every visible card's
// text and easily outsizes a real description) — exclude both.
const blocks = Array.from(document.querySelectorAll("div"))
.filter(
(el) =>
el.children.length <= 3 && !el.querySelector("style, script, .EimVGf"),
)
.map((el) => clean(el.textContent))
.filter((text) => text.length > 200);
blocks.sort((a, b) => b.length - a.length);
return {
applyLinks,
description: blocks[0]?.slice(0, 5_000),
};
}
async function openCardAndExtractDetail(
page: Page,
selectors: readonly string[],
index: number,
docId: string | undefined,
): Promise<ScrapedDetail | null> {
const elementHandle = await page.evaluateHandle(
(args: { selectors: readonly string[]; idx: number }) => {
for (const selector of args.selectors) {
const items = document.querySelectorAll(selector);
if (items.length > 0) return items[args.idx] ?? null;
}
return null;
},
{ selectors, idx: index },
);
const element = elementHandle.asElement();
if (!element) return null;
await element.scrollIntoViewIfNeeded();
// A single click is unreliable here: clicking a card while a *different*
// card's detail pane is already open frequently just closes the open pane
// instead of switching to the new one (confirmed by watching real runs —
// it alternates open/closed rather than opening the newly clicked card).
// A double-click reopens reliably; verified across 6 sequential cards.
await element.dblclick({ timeout: 5_000, delay: 80 });
if (docId) {
try {
await page.waitForSelector(`[data-encoded-docid="${docId}"]`, {
timeout: 6_000,
});
} catch {
// fall through — extractDetailScript falls back to a document-wide
// scan for apply links if the docid-scoped block never shows up.
}
}
await page.waitForTimeout(500);
return page.evaluate(extractDetailScript, docId);
}
async function saveDebugArtifacts(page: Page): Promise<void> {
try {
await mkdir(dirname(DEBUG_HTML_PATH), { recursive: true });
await writeFile(DEBUG_HTML_PATH, await page.content(), "utf-8");
await page.screenshot({ path: DEBUG_SCREENSHOT_PATH, fullPage: true });
} catch (error) {
console.error(
`[google-jobs] failed to save debug artifacts: ${error instanceof Error ? error.message : String(error)}`,
);
}
}
async function scrapeTerm(
context: BrowserContext,
term: string,
remainingCap: number,
options: { headless: boolean; unblockTimeoutMs: number },
): Promise<ScrapedJob[]> {
const page = await context.newPage();
const out: ScrapedJob[] = [];
try {
await page.goto(buildSearchUrl(term), {
waitUntil: "domcontentloaded",
timeout: 45_000,
});
await dismissConsent(page);
await page.waitForTimeout(1_500);
if (await isBlockedPage(page)) {
const solvedByHuman =
!options.headless &&
(await waitForHumanToUnblock(page, options.unblockTimeoutMs));
if (!solvedByHuman) {
throw new GoogleBlockedError(
"Google blocked this request (CAPTCHA / unusual traffic detection). This is common from shared, VPN, or datacenter IPs. Try again later, reduce run frequency, run from a residential network, set GOOGLE_JOBS_PROXY, or set GOOGLE_JOBS_HEADLESS=false to solve it yourself.",
);
}
}
const listReady = await waitForJobsList(page);
if (!listReady) {
if (await isEmptyJobsVertical(page)) {
console.error(
`[google-jobs] Google's Jobs tab reported no matches for "${term}" (not a block or selector issue — try a broader/more common phrasing).`,
);
} else {
console.error(
`[google-jobs] Jobs list did not appear for "${term}" (url: ${page.url()}). Saved a screenshot + HTML dump to ${DEBUG_SCREENSHOT_PATH} for inspection.`,
);
await saveDebugArtifacts(page);
}
return out;
}
await growJobsList(page, CARD_SELECTORS, remainingCap);
const cards = await page.evaluate(extractCardsScript, CARD_SELECTORS);
const limit = Math.min(cards.length, remainingCap);
for (let index = 0; index < limit; index += 1) {
const card = cards[index];
// Google's own permalink into this specific listing — always available
// from the list view, so results stay unique/useful even if the detail
// click below fails for this card.
const shareUrl = card.shareUrl;
try {
const detail = await openCardAndExtractDetail(
page,
CARD_SELECTORS,
index,
card.docId,
);
const applyLink = detail?.applyLinks[0];
const jobUrl = shareUrl ?? applyLink?.href ?? page.url();
out.push({
title: card.title,
employer: card.employer,
jobUrl,
applicationLink: applyLink?.href,
location: card.location,
jobDescription: detail?.description,
postedVia: card.via,
isRemote: /remote/i.test(`${card.location ?? ""} ${card.title}`),
});
} catch (error) {
console.error(
`[google-jobs] card ${index} for "${term}" failed: ${
error instanceof Error ? error.message : String(error)
}`,
);
if (shareUrl) {
// Detail click failed, but we still have a unique, valid link to
// the listing from the card itself — don't drop the job entirely.
out.push({
title: card.title,
employer: card.employer,
jobUrl: shareUrl,
location: card.location,
postedVia: card.via,
isRemote: /remote/i.test(`${card.location ?? ""} ${card.title}`),
});
}
}
await page.waitForTimeout(300 + Math.floor(Math.random() * 400));
}
} finally {
await page.close();
}
return out;
}
async function main(): Promise<void> {
const terms = parseTerms(process.env.GOOGLE_JOBS_SEARCH_TERMS);
const maxJobsEnv = Number.parseInt(
process.env.GOOGLE_JOBS_MAX_JOBS ?? "30",
10,
);
const cap = Number.isFinite(maxJobsEnv)
? Math.max(1, Math.min(maxJobsEnv, 150))
: 30;
// Headed mode lets a human solve a CAPTCHA in the visible window; only
// useful when run directly on a machine with a display, not on a headless
// cron/server pipeline.
const headless = process.env.GOOGLE_JOBS_HEADLESS !== "false";
const unblockTimeoutMs = Number.parseInt(
process.env.GOOGLE_JOBS_UNBLOCK_TIMEOUT_MS ?? "300000",
10,
);
const browser = await launchBrowser(headless);
const hasSavedState = await access(STATE_PATH)
.then(() => true)
.catch(() => false);
const context = await browser.newContext(
hasSavedState ? { storageState: STATE_PATH } : {},
);
// tsx/esbuild compiles this file with `keepNames`, which injects
// `__name(fn, "fn")` calls into any named function it transpiles —
// including ones later serialized into `page.evaluate()` scripts via
// `Function.prototype.toString()`. That helper only exists in our Node
// process, not in the browser page, so without this no-op polyfill any
// evaluate script with a nested named function throws
// `ReferenceError: __name is not defined`.
await context.addInitScript(() => {
(window as unknown as { __name?: (fn: unknown) => unknown }).__name = (
fn: unknown,
) => fn;
});
const all: ScrapedJob[] = [];
const seenUrls = new Set<string>();
try {
for (const term of terms) {
if (all.length >= cap) break;
try {
const rows = await scrapeTerm(context, term, cap - all.length, {
headless,
unblockTimeoutMs,
});
for (const row of rows) {
if (all.length >= cap) break;
if (seenUrls.has(row.jobUrl)) continue;
seenUrls.add(row.jobUrl);
all.push(row);
}
} catch (error) {
if (error instanceof GoogleBlockedError) {
// IP-level block affects the whole run; stop trying further terms
// and let it propagate so the process exits non-zero with a clear message.
throw error;
}
console.error(
`[google-jobs] term "${term}" failed: ${
error instanceof Error ? error.message : String(error)
}`,
);
}
}
} finally {
// Persist cookies/session (even a partially solved one) so future
// headless/automated runs can reuse it and hit fewer CAPTCHA walls.
await mkdir(dirname(STATE_PATH), { recursive: true });
await context.storageState({ path: STATE_PATH }).catch(() => undefined);
await browser.close();
}
await mkdir(dirname(OUTPUT_PATH), { recursive: true });
await writeFile(OUTPUT_PATH, JSON.stringify(all, null, 2), "utf-8");
}
main().catch((error) => {
console.error(error instanceof Error ? error.message : String(error));
process.exit(1);
});

View File

@ -0,0 +1,154 @@
import { spawn, spawnSync } from "node:child_process";
import { readFile } from "node:fs/promises";
import { createRequire } from "node:module";
import { dirname, join } from "node:path";
import { fileURLToPath } from "node:url";
import { envForExtractorSubprocess } from "@shared/extractor-subprocess-env.js";
import type { CreateJobInput } from "@shared/types/jobs";
const srcDir = dirname(fileURLToPath(import.meta.url));
const EXTRACTOR_DIR = join(srcDir, "..");
const OUTPUT_PATH = join(EXTRACTOR_DIR, "storage/jobs.json");
const require = createRequire(import.meta.url);
const TSX_CLI_PATH = (() => {
try {
return require.resolve("tsx/dist/cli.mjs");
} catch {
return null;
}
})();
/** Browser scrape can take a while (per-term navigation + per-card clicks). */
const SUBPROCESS_TIMEOUT_MS = 4 * 60 * 1000;
interface ScrapedJob {
title?: string;
employer?: string;
jobUrl?: string;
applicationLink?: string;
location?: string;
jobDescription?: string;
postedVia?: string;
isRemote?: boolean;
}
export interface RunGoogleJobsOptions {
searchTerms?: string[];
maxJobs?: number;
}
export interface GoogleJobsResult {
success: boolean;
jobs: CreateJobInput[];
error?: string;
}
function canRunNpmCommand(): boolean {
const result = spawnSync("npm", ["--version"], { stdio: "ignore" });
return !result.error && result.status === 0;
}
function mapJob(row: ScrapedJob): CreateJobInput | null {
const jobUrl = row.jobUrl?.trim();
if (!jobUrl) return null;
return {
source: "google-jobs",
title: row.title?.trim() || "Unknown Title",
employer: row.employer?.trim() || "Unknown Employer",
jobUrl,
applicationLink: row.applicationLink?.trim() || jobUrl,
location: row.location?.trim() || undefined,
jobDescription: row.jobDescription?.trim() || undefined,
isRemote: row.isRemote ?? undefined,
};
}
export async function runGoogleJobs(
options: RunGoogleJobsOptions = {},
): Promise<GoogleJobsResult> {
const searchTerms =
options.searchTerms && options.searchTerms.length > 0
? options.searchTerms
: ["software engineer"];
const maxJobs = options.maxJobs ?? 30;
const useNpmCommand = canRunNpmCommand();
if (!TSX_CLI_PATH && !useNpmCommand) {
return {
success: false,
jobs: [],
error: "Unable to execute Google Jobs extractor (npm/tsx unavailable)",
};
}
try {
await new Promise<void>((resolve, reject) => {
const extractorEnv = envForExtractorSubprocess({
...process.env,
GOOGLE_JOBS_SEARCH_TERMS: JSON.stringify(searchTerms),
GOOGLE_JOBS_MAX_JOBS: String(maxJobs),
});
const child = TSX_CLI_PATH
? spawn(process.execPath, [TSX_CLI_PATH, join(srcDir, "main.ts")], {
cwd: EXTRACTOR_DIR,
env: extractorEnv,
stdio: ["ignore", "pipe", "pipe"],
})
: spawn("npx", ["tsx", join(srcDir, "main.ts")], {
cwd: EXTRACTOR_DIR,
env: extractorEnv,
stdio: ["ignore", "pipe", "pipe"],
});
let stderr = "";
child.stderr?.on("data", (chunk: Buffer) => {
stderr += chunk.toString();
});
const timeout = setTimeout(() => {
child.kill("SIGKILL");
reject(new Error("Google Jobs browser scrape timed out"));
}, SUBPROCESS_TIMEOUT_MS);
child.on("error", (error) => {
clearTimeout(timeout);
reject(error);
});
child.on("close", (code) => {
clearTimeout(timeout);
if (code === 0) {
resolve();
return;
}
reject(
new Error(
stderr.trim() ||
`Google Jobs browser scrape exited with code ${code}`,
),
);
});
});
const raw = await readFile(OUTPUT_PATH, "utf-8");
const parsed = JSON.parse(raw) as unknown;
const rows = Array.isArray(parsed) ? parsed : [];
const jobs: CreateJobInput[] = [];
const seen = new Set<string>();
for (const value of rows) {
if (!value || typeof value !== "object" || Array.isArray(value)) continue;
const mapped = mapJob(value as ScrapedJob);
if (!mapped) continue;
const key = mapped.jobUrl;
if (seen.has(key)) continue;
seen.add(key);
jobs.push(mapped);
}
return { success: true, jobs };
} catch (error) {
const message = error instanceof Error ? error.message : "Unknown error";
return { success: false, jobs: [], error: message };
}
}

View File

@ -0,0 +1,17 @@
{
"compilerOptions": {
"module": "ESNext",
"moduleResolution": "bundler",
"target": "ES2022",
"outDir": "dist",
"strict": true,
"noUnusedLocals": false,
"lib": ["ES2022", "DOM"],
"types": ["node"],
"baseUrl": ".",
"paths": {
"@shared/*": ["../../shared/src/*"]
}
},
"include": ["./manifest.ts", "./src/**/*"]
}

View File

@ -116,11 +116,32 @@ router.addHandler(
const title = (await titleLocator.textContent())?.trim() ?? null;
const jobUrl = toAbsolute(await titleLocator.getAttribute("href"));
const employerImg = article.locator("figure img");
const employer = (await employerImg.getAttribute("alt"))?.trim() ?? null;
const employerAnchor = article.locator("figure a");
const employerUrl = toAbsolute(await employerAnchor.getAttribute("href"));
// Employer logo/link are optional — some cards omit <figure img>, and a
// hard getAttribute timeout used to fail the whole region page (0 jobs)
// even after we had already counted cards.
let employer: string | null = null;
let employerUrl: string | null = null;
try {
const employerImg = article.locator("figure img").first();
if ((await employerImg.count()) > 0) {
employer =
(
await employerImg.getAttribute("alt", { timeout: 2000 })
)?.trim() ?? null;
}
} catch {
// optional
}
try {
const employerAnchor = article.locator("figure a").first();
if ((await employerAnchor.count()) > 0) {
employerUrl = toAbsolute(
await employerAnchor.getAttribute("href", { timeout: 2000 }),
);
}
} catch {
// optional
}
let disciplines: string | null = null;
try {

View File

@ -14,8 +14,10 @@ import type {
import type { CreateJobInput } from "@shared/types/jobs";
const API_URL = "https://himalayas.app/jobs/api";
const PAGE_SIZE = 50;
const MAX_PAGES = 5;
/** Himalayas silently caps `limit` at 20; requesting more makes us think the
* feed ended early (`raw.length < PAGE_SIZE`) and stop after the first page. */
const PAGE_SIZE = 20;
const MAX_PAGES = 10;
interface HimalayasJob {
title?: string;

View File

@ -1,3 +1,8 @@
# python-jobspy requires Python 3.10+ (wheels not published for 3.9 and below).
python-jobspy
#
# Pinned to 1.1.82: scrape_jobs.py monkeypatches Glassdoor._fetch_jobs_page
# to work around an upstream bug where any partial GraphQL error (even for
# unrelated fields) discards a whole page of valid job data. If bumping this
# version, re-check that method still matches before removing the pin.
python-jobspy==1.1.82
pandas

View File

@ -1,12 +1,81 @@
import csv
import json
import logging
import os
from concurrent.futures import ThreadPoolExecutor, as_completed
from pathlib import Path
import pandas as pd
from jobspy import scrape_jobs
from jobspy.exception import GlassdoorException
from jobspy.glassdoor import Glassdoor
from jobspy.glassdoor.util import get_cursor_for_page
PROGRESS_PREFIX = "JOBOPS_PROGRESS "
_glassdoor_log = logging.getLogger("JobSpy:Glassdoor")
def _patched_glassdoor_fetch_jobs_page(
self, scraper_input, location_id, location_type, page_num, cursor
):
"""Replaces python-jobspy's Glassdoor._fetch_jobs_page (as of 1.1.82).
Upstream treats *any* `errors` entry in Glassdoor's GraphQL response as
fatal and discards the whole page, even when the job listings we need
parsed fine. In practice Glassdoor's backend intermittently (but
consistently, as of writing) returns a partial error for an unrelated
field observed: `jobsPageSeoData` failing with
`SUBREQUEST_HTTP_ERROR` / "dns error" on Glassdoor's own infra — while
`data.jobListings.jobListings` still comes back with real results. The
upstream check throws that good data away, so every Glassdoor run
returns 0 jobs. This patch only bails out when the job listings payload
itself is actually missing.
"""
jobs = []
self.scraper_input = scraper_input
try:
payload = self._add_payload(location_id, location_type, page_num, cursor)
response = self.session.post(
f"{self.base_url}/graph",
timeout_seconds=15,
data=payload,
)
if response.status_code != 200:
raise GlassdoorException(
f"bad response status code: {response.status_code}"
)
res_json = response.json()[0]
job_listings = res_json.get("data", {}).get("jobListings")
if job_listings is None:
raise ValueError("Error encountered in API response")
if "errors" in res_json:
_glassdoor_log.warning(
"Glassdoor GraphQL returned partial errors for unrelated "
"fields; continuing with the job listings that did parse: %s",
res_json["errors"],
)
except (GlassdoorException, ValueError, Exception) as e:
_glassdoor_log.error(f"Glassdoor: {str(e)}")
return jobs, None
jobs_data = job_listings["jobListings"]
with ThreadPoolExecutor(max_workers=self.jobs_per_page) as executor:
future_to_job_data = {
executor.submit(self._process_job, job): job for job in jobs_data
}
for future in as_completed(future_to_job_data):
try:
job_post = future.result()
if job_post:
jobs.append(job_post)
except Exception as exc:
raise GlassdoorException(f"Glassdoor generated an exception: {exc}")
return jobs, get_cursor_for_page(job_listings["paginationCursors"], page_num + 1)
Glassdoor._fetch_jobs_page = _patched_glassdoor_fetch_jobs_page
COUNTRY_ALIASES = {
"uk": "united kingdom",
"united kingdom": "united kingdom",

View File

@ -394,15 +394,15 @@ export async function activateSearchProfileForBasicAuthUser(
}
if (!listPayload.ok || !Array.isArray(listPayload.data)) return;
const normalizedUsername = username.trim().toLowerCase();
const match = listPayload.data.find((row) => {
const matches = listPayload.data.filter((row) => {
const aliases = (row.data?.basicAuthUser ?? "")
.split(",")
.map((part) => part.trim().toLowerCase())
.filter(Boolean);
return aliases.includes(normalizedUsername);
});
if (!match) return;
await fetch(`${API_BASE}/profiles/${match.id}/activate`, {
if (matches.length !== 1) return;
await fetch(`${API_BASE}/profiles/${matches[0].id}/activate`, {
method: "POST",
headers: {
"Content-Type": "application/json",

View File

@ -1,8 +1,17 @@
import { PRIOR_SKIP_DISMISS_LOOKBACK_MS } from "@shared/job-fingerprint";
import { createJob } from "@shared/testing/factories";
import { describe, expect, it } from "vitest";
import { buildDuplicateDismissHints } from "./job-dedup";
describe("buildDuplicateDismissHints", () => {
const now = Date.parse("2026-06-10T12:00:00.000Z");
const recentUpdatedAt = new Date(
now - PRIOR_SKIP_DISMISS_LOOKBACK_MS + 86_400_000,
).toISOString();
const staleUpdatedAt = new Date(
now - PRIOR_SKIP_DISMISS_LOOKBACK_MS - 86_400_000,
).toISOString();
it("flags open jobs that match a skipped posting", () => {
const jobs = [
createJob({
@ -10,6 +19,7 @@ describe("buildDuplicateDismissHints", () => {
employer: "Acme",
title: "SDET",
status: "skipped",
updatedAt: recentUpdatedAt,
}),
createJob({
id: "open-1",
@ -25,8 +35,71 @@ describe("buildDuplicateDismissHints", () => {
}),
];
const hints = buildDuplicateDismissHints(jobs);
const hints = buildDuplicateDismissHints(jobs, now);
expect(hints.get("open-1")).toBe("skipped");
expect(hints.has("open-2")).toBe(false);
});
it("matches employer prefix and plural title variants", () => {
const jobs = [
createJob({
id: "skipped-1",
employer: "CGI",
title: "Automation Test Engineer",
status: "skipped",
updatedAt: recentUpdatedAt,
}),
createJob({
id: "open-1",
employer: ". CGI IT UK Limited",
title: "Automation Test Engineers",
status: "discovered",
}),
];
const hints = buildDuplicateDismissHints(jobs, now);
expect(hints.get("open-1")).toBe("skipped");
});
it("ignores skipped jobs older than the lookback window", () => {
const jobs = [
createJob({
id: "skipped-1",
employer: "Acme",
title: "SDET",
status: "skipped",
updatedAt: staleUpdatedAt,
}),
createJob({
id: "open-1",
employer: "Acme",
title: "SDET",
status: "discovered",
}),
];
const hints = buildDuplicateDismissHints(jobs, now);
expect(hints.has("open-1")).toBe(false);
});
it("does not match different titles at the same employer", () => {
const jobs = [
createJob({
id: "skipped-1",
employer: "Cognizant",
title: "Automation Test Engineer",
status: "skipped",
updatedAt: recentUpdatedAt,
}),
createJob({
id: "open-1",
employer: "Cognizant",
title: "AI Automation Test Engineer",
status: "discovered",
}),
];
const hints = buildDuplicateDismissHints(jobs, now);
expect(hints.has("open-1")).toBe(false);
});
});

View File

@ -1,44 +1,60 @@
import { collectJobDedupKeys } from "@shared/job-fingerprint";
import {
jobsMatchEmployerAndTitleDismiss,
PRIOR_SKIP_DISMISS_LOOKBACK_MS,
} from "@shared/job-fingerprint";
import type { JobListItem, JobStatus } from "@shared/types";
export type DuplicateDismissReason = "skipped" | "applied";
const OPEN_JOB_STATUSES = new Set<JobStatus>([
"discovered",
"ready",
"processing",
]);
function isWithinDismissLookback(updatedAt: string, nowMs: number): boolean {
const updatedMs = Date.parse(updatedAt);
if (!Number.isFinite(updatedMs)) return false;
return nowMs - updatedMs <= PRIOR_SKIP_DISMISS_LOOKBACK_MS;
}
/**
* Map open jobs to a prior skip/apply when employer+title or description matches.
* Map open jobs to a prior skip/apply when the same company and title match
* within the last 90 days (see {@link PRIOR_SKIP_DISMISS_LOOKBACK_MS}).
*/
export function buildDuplicateDismissHints(
jobs: readonly JobListItem[],
nowMs: number = Date.now(),
): Map<string, DuplicateDismissReason> {
const dismissedKeys = new Map<string, DuplicateDismissReason>();
const dismissed: Array<{
employer: string;
title: string;
reason: DuplicateDismissReason;
}> = [];
for (const job of jobs) {
if (job.status !== "skipped" && job.status !== "applied") continue;
const reason: DuplicateDismissReason =
job.status === "applied" ? "applied" : "skipped";
for (const key of collectJobDedupKeys({
if (!isWithinDismissLookback(job.updatedAt, nowMs)) continue;
dismissed.push({
employer: job.employer,
title: job.title,
})) {
if (!dismissedKeys.has(key)) dismissedKeys.set(key, reason);
}
reason: job.status === "applied" ? "applied" : "skipped",
});
}
const hints = new Map<string, DuplicateDismissReason>();
const openStatuses = new Set<JobStatus>([
"discovered",
"ready",
"processing",
]);
for (const job of jobs) {
if (!openStatuses.has(job.status)) continue;
for (const key of collectJobDedupKeys({
employer: job.employer,
title: job.title,
})) {
const reason = dismissedKeys.get(key);
if (reason) {
hints.set(job.id, reason);
if (!OPEN_JOB_STATUSES.has(job.status)) continue;
for (const prior of dismissed) {
if (
jobsMatchEmployerAndTitleDismiss({
employerA: job.employer,
titleA: job.title,
employerB: prior.employer,
titleB: prior.title,
})
) {
hints.set(job.id, prior.reason);
break;
}
}
@ -46,5 +62,3 @@ export function buildDuplicateDismissHints(
return hints;
}
export { collectJobDedupKeys };

View File

@ -63,6 +63,8 @@ export const OrchestratorPage: React.FC = () => {
setEmployerFilterTokens,
applySettingsCompanySkipList,
setApplySettingsCompanySkipList,
hidePriorSkips,
setHidePriorSkips,
sort,
setSort,
resetFilters,
@ -173,8 +175,8 @@ export const OrchestratorPage: React.FC = () => {
);
const duplicateDismissHints = useMemo(
() => buildDuplicateDismissHints(jobs),
[jobs],
() => (hidePriorSkips ? buildDuplicateDismissHints(jobs) : undefined),
[hidePriorSkips, jobs],
);
const jobListFilterExtras = useMemo(
@ -531,6 +533,8 @@ export const OrchestratorPage: React.FC = () => {
onApplySettingsCompanySkipListChange={
setApplySettingsCompanySkipList
}
hidePriorSkips={hidePriorSkips}
onHidePriorSkipsChange={setHidePriorSkips}
sponsorFilter={sponsorFilter}
onSponsorFilterChange={setSponsorFilter}
sponsorshipSignalsFilter={sponsorshipSignalsFilter}

View File

@ -640,6 +640,7 @@ export const AutomaticRunTab: React.FC<AutomaticRunTabProps> = ({
</CardHeader>
<CardContent className="space-y-4">
<KeywordSetPicker
key={settings?.activeProfileId ?? "default"}
disabled={isSaving || isPipelineRunning}
onActiveSetChange={handleActiveKeywordSetChange}
/>

View File

@ -53,6 +53,8 @@ const renderFilters = (overrides?: Partial<FiltersProps>) => {
onEmployerIncludeChange: vi.fn(),
onEmployerExcludeChange: vi.fn(),
applySettingsCompanySkipList: true,
hidePriorSkips: true,
onHidePriorSkipsChange: vi.fn(),
onApplySettingsCompanySkipListChange: vi.fn(),
sponsorFilter: "all" as SponsorFilter,
onSponsorFilterChange: vi.fn(),

View File

@ -69,6 +69,8 @@ interface OrchestratorFiltersProps {
onEmployerExcludeChange: (values: string[]) => void;
applySettingsCompanySkipList: boolean;
onApplySettingsCompanySkipListChange: (value: boolean) => void;
hidePriorSkips: boolean;
onHidePriorSkipsChange: (value: boolean) => void;
sponsorFilter: SponsorFilter;
onSponsorFilterChange: (value: SponsorFilter) => void;
sponsorshipSignalsFilter: SponsorshipSignalId[];
@ -183,6 +185,8 @@ export const OrchestratorFilters: React.FC<OrchestratorFiltersProps> = ({
onEmployerExcludeChange,
applySettingsCompanySkipList,
onApplySettingsCompanySkipListChange,
hidePriorSkips,
onHidePriorSkipsChange,
sponsorFilter,
onSponsorFilterChange,
sponsorshipSignalsFilter,
@ -253,6 +257,7 @@ export const OrchestratorFilters: React.FC<OrchestratorFiltersProps> = ({
employerIncludeFilter.length > 0 || employerExcludeFilter.length > 0,
) +
Number(!applySettingsCompanySkipList) +
Number(!hidePriorSkips) +
Number(sponsorFilter !== "all") +
Number(sponsorshipSignalsFilter.length > 0) +
Number(hideSponsorBlockers) +
@ -271,6 +276,7 @@ export const OrchestratorFilters: React.FC<OrchestratorFiltersProps> = ({
employerIncludeFilter.length,
employerExcludeFilter.length,
applySettingsCompanySkipList,
hidePriorSkips,
sponsorFilter,
sponsorshipSignalsFilter.length,
hideSponsorBlockers,
@ -542,6 +548,29 @@ export const OrchestratorFilters: React.FC<OrchestratorFiltersProps> = ({
</div>
</div>
<div className="flex items-start gap-3 rounded-md border border-border/60 bg-muted/20 px-3 py-3">
<Checkbox
id="hide-prior-skips"
checked={hidePriorSkips}
onCheckedChange={(checked) => {
onHidePriorSkipsChange(checked === true);
}}
/>
<div className="space-y-1">
<label
htmlFor="hide-prior-skips"
className="text-sm font-medium leading-none cursor-pointer"
>
Hide roles you already skipped
</label>
<p className="text-xs text-muted-foreground">
Hides Discovered and Ready rows that match a skipped
or applied job with the same company and title from
the last 90 days.
</p>
</div>
</div>
<TokenizedInput
id="employer-include-filter"
values={employerIncludeFilter}

View File

@ -5,7 +5,7 @@ import type { SearchProfile } from "@shared/types";
import { useMutation, useQuery, useQueryClient } from "@tanstack/react-query";
import { Loader2 } from "lucide-react";
import type React from "react";
import { useMemo } from "react";
import { useEffect, useMemo, useState } from "react";
import { toast } from "sonner";
import { Button } from "@/components/ui/button";
import {
@ -27,17 +27,38 @@ export const ProfileQuickSwitch: React.FC = () => {
const { data: profiles = [], isLoading } = useQuery<SearchProfile[]>({
queryKey: queryKeys.searchProfiles.list(),
queryFn: api.listProfiles,
select: (rows) => {
const seen = new Set<string>();
return rows.filter((profile) => {
if (seen.has(profile.id)) return false;
seen.add(profile.id);
return true;
});
},
});
const [selectedId, setSelectedId] = useState<string | null>(null);
const activeId = useMemo(() => {
if (selectedId && profiles.some((profile) => profile.id === selectedId)) {
return selectedId;
}
const sid = settings?.activeProfileId?.trim() ?? "";
if (sid && profiles.some((p) => p.id === sid)) return sid;
if (sid && profiles.some((profile) => profile.id === sid)) return sid;
return profiles[0]?.id ?? "";
}, [selectedId, settings?.activeProfileId, profiles]);
useEffect(() => {
const sid = settings?.activeProfileId?.trim() ?? "";
if (sid && profiles.some((profile) => profile.id === sid)) {
setSelectedId(sid);
}
}, [settings?.activeProfileId, profiles]);
const activateMutation = useMutation({
mutationFn: (id: string) => api.activateProfile(id),
onSuccess: async () => {
onSuccess: async (_data, profileId) => {
setSelectedId(profileId);
await Promise.all([
queryClient.invalidateQueries({ queryKey: queryKeys.jobs.all }),
queryClient.invalidateQueries({ queryKey: queryKeys.profile.all }),
@ -46,7 +67,10 @@ export const ProfileQuickSwitch: React.FC = () => {
}),
]);
await refreshSettings();
toast.success("Search profile updated");
const profileName =
profiles.find((profile) => profile.id === profileId)?.name ??
"Search profile";
toast.success(`${profileName} active`);
},
onError: (err: unknown) => {
const message =

View File

@ -199,6 +199,7 @@ export function buildMaxCoverageDiscoveryLimits(args: {
builtinMaxJobsPerTerm: boardCap,
builtinMaxPagesPerTerm: 10,
wellfoundMaxJobsPerTerm: Math.min(100, boardCap),
googleJobsMaxJobsPerTerm: Math.min(30, boardCap),
qajobsboardMaxJobsPerTerm: boardCap,
arcMaxJobsPerPath: boardCap,
smartrecruitersMaxJobsPerCompany: boardCap,

View File

@ -435,6 +435,11 @@ export const useOrchestratorFilters = () => {
[searchParams],
);
const hidePriorSkips = useMemo(
() => searchParams.get("hidePriorSkips") !== "0",
[searchParams],
);
const setApplySettingsCompanySkipList = useCallback(
(value: boolean) => {
setSearchParams(
@ -449,6 +454,20 @@ export const useOrchestratorFilters = () => {
[setSearchParams],
);
const setHidePriorSkips = useCallback(
(value: boolean) => {
setSearchParams(
(prev) => {
if (value) prev.delete("hidePriorSkips");
else prev.set("hidePriorSkips", "0");
return prev;
},
{ replace: true },
);
},
[setSearchParams],
);
const sort = useMemo((): JobSort => {
const sortValue = searchParams.get("sort");
if (!sortValue) return DEFAULT_SORT;
@ -508,6 +527,7 @@ export const useOrchestratorFilters = () => {
prev.delete("employer");
prev.delete("employerExclude");
prev.delete("skipList");
prev.delete("hidePriorSkips");
return prev;
},
{ replace: true },
@ -540,6 +560,8 @@ export const useOrchestratorFilters = () => {
setEmployerFilterTokens,
applySettingsCompanySkipList,
setApplySettingsCompanySkipList,
hidePriorSkips,
setHidePriorSkips,
sort,
setSort,
resetFilters,

View File

@ -34,6 +34,7 @@ describe("orchestrator utils", () => {
"testdevjobs",
"wellfound",
"builtin",
"google-jobs",
] as const) {
expect(enabled).toContain(source);
}

View File

@ -334,6 +334,7 @@ export const getEnabledSources = (
source === "testdevjobs" ||
source === "wellfound" ||
source === "builtin" ||
source === "google-jobs" ||
source === "qajobsboard" ||
source === "arcdev"
) {

View File

@ -104,8 +104,8 @@ export const JobSourcesSettingsSection: React.FC<
<p className="text-sm text-muted-foreground">
Configure company slugs, tenant IDs, and career-page URLs so ATS and
direct employer sources appear in the pipeline run dialog. Public job
boards (Working Nomads, TestDevJobs, Wellfound, etc.) need no setup
here.
boards (Working Nomads, TestDevJobs, Wellfound, Google Jobs, etc.)
need no setup here.
</p>
<div className="space-y-4">

View File

@ -175,6 +175,10 @@ profilesRouter.post(
const basicUser = parseBasicAuthUsername(req.headers.authorization)?.trim();
if (isBasicAuthEnabled() && basicUser) {
await profilesRepo.setActiveSearchProfileIdForBasicAuthUser(
basicUser,
profile.id,
);
clearProfileCache();
return ok(res, { activated: true, profileId: profile.id });
}

View File

@ -271,6 +271,7 @@ export const DEMO_SOURCE_BASE_URLS: Record<JobSource, string> = {
testdevjobs: "https://testdevjobs.com",
wellfound: "https://wellfound.com",
builtin: "https://builtin.com",
"google-jobs": "https://www.google.com/search?ibp=htl;jobs",
fourdayweek: "https://4dayweek.io",
ashby: "https://jobs.ashbyhq.com",
lever: "https://jobs.lever.co",

View File

@ -29,6 +29,75 @@ function sqlInsertSearchProfileSeed(input: {
return `INSERT OR IGNORE INTO search_profiles (id, name, data, created_at, updated_at) VALUES ('${id}', '${name}', '${data}', datetime('now'), datetime('now'))`;
}
function sqlJsonLiteral(values: string[]): string {
return JSON.stringify(values).replace(/'/g, "''");
}
/** Canada / remote — SDET & QA automation title variants for discovery + scoring. */
const ILIA_DOBKIN_CA_TARGET_ROLES = [
"SDET",
"Senior SDET",
"Lead SDET",
"Staff SDET",
"Principal SDET",
"Software Development Engineer in Test",
"Senior Software Development Engineer in Test",
"Software Engineer in Test",
"Software Test Engineer",
"Senior Software Test Engineer",
"QA Automation Engineer",
"Senior QA Automation Engineer",
"Quality Assurance Automation Engineer",
"Test Automation Engineer",
"Senior Test Automation Engineer",
"Automation Test Engineer",
"Automation Engineer",
"QA Engineer",
"Senior QA Engineer",
"Quality Engineer",
"Test Automation Lead",
"QA Automation Lead",
"Test Lead",
];
/** Job-board search terms — title variants plus common stack/tool pairings. */
const ILIA_DOBKIN_CA_KEYWORD_TERMS = [
...ILIA_DOBKIN_CA_TARGET_ROLES.map((role) => role.toLowerCase()),
"playwright SDET",
"SDET playwright",
"SDET java",
"SDET typescript",
"SDET cypress",
"SDET selenium",
"SDET api testing",
"SDET remote",
"QA automation remote",
"contract SDET",
"contract QA automation",
"SET",
"software engineer test",
];
const ILIA_DOBKIN_US_TARGET_ROLES = [
...ILIA_DOBKIN_CA_TARGET_ROLES,
"Computer Systems Analyst",
"Quality Assurance Engineer",
];
const ILIA_DOBKIN_US_KEYWORD_TERMS = [
...ILIA_DOBKIN_CA_KEYWORD_TERMS,
"computer systems analyst",
"systems analyst",
"computer systems analyst QA",
"SDET United States",
"SDET USA",
"SDET remote US",
"QA automation engineer USA",
"test automation engineer United States",
"quality assurance engineer automation",
"TN SDET",
];
const migrations = [
`CREATE TABLE IF NOT EXISTS jobs (
id TEXT PRIMARY KEY,
@ -817,28 +886,40 @@ const migrations = [
id: "685b0000-0000-4000-8000-000000000001",
name: "Ilia Dobkin",
profile: {
targetRoles: [
"Software Development Engineer in Test",
"QA Automation Engineer",
"SDET",
],
targetRoles: ILIA_DOBKIN_CA_TARGET_ROLES,
experienceLevel: "Senior",
mustHaveSkills: [
"Playwright",
"Cypress",
"Selenium",
"TypeScript",
"Java",
"API testing",
"CI/CD",
"Selenium",
"Cypress",
],
niceToHaveSkills: [
"Spring Boot",
"Python",
".NET",
"Contract testing",
"PostgreSQL",
"JUnit",
],
niceToHaveSkills: ["Python", ".NET", "CI/CD"],
dealBreakers: [],
preferredWorkArrangement: ["remote", "hybrid"],
preferredLocations: ["Toronto", "Ontario", "Canada"],
preferredLocations: [
"Toronto",
"Ontario",
"GTA",
"Greater Toronto Area",
"Canada",
"Remote Canada",
],
minimumSalary: "",
industriesToTarget: ["Software", "FinTech", "iGaming"],
industriesToTarget: ["Software", "FinTech", "iGaming", "Enterprise SaaS"],
industriesToAvoid: [],
aboutMe:
"Ilia Dobkin — SDET / test automation; paired login user `ilia` activates this profile and local resume ilia-dobkin.json (unset JOBOPS_LOCAL_RESUME_PATH so Settings path applies).",
"Ilia Dobkin — Senior SDET / test automation (Canada & remote). Paired login user `ilia` activates this profile and local resume ilia-dobkin.json (unset JOBOPS_LOCAL_RESUME_PATH so Settings path applies).",
basicAuthUser: "ilia,dobkin",
resumeLocalPath: "../data/resumes/ilia-dobkin.json",
},
@ -867,7 +948,7 @@ const migrations = [
'ilia-dobkin-keyword-set',
'685b0000-0000-4000-8000-000000000001',
'SDET',
'["SDET","QA automation engineer","automation test engineer","playwright","software development engineer in test","quality assurance automation engineer"]',
'${sqlJsonLiteral(ILIA_DOBKIN_CA_KEYWORD_TERMS)}',
1,
datetime('now'),
datetime('now')
@ -976,6 +1057,91 @@ const migrations = [
updated_at = datetime('now')
WHERE id = '685b0000-0000-4000-8000-000000000002'
AND coalesce(json_extract(data, '$.resumeLocalPath'), '') = ''`,
sqlInsertSearchProfileSeed({
id: "685b0000-0000-4000-8000-000000000003",
name: "Ilia Dobkin (US)",
profile: {
targetRoles: ILIA_DOBKIN_US_TARGET_ROLES,
experienceLevel: "Senior",
mustHaveSkills: [
"Playwright",
"TypeScript",
"API testing",
"CI/CD",
"Java",
"Integration testing",
"Selenium",
],
niceToHaveSkills: [
"Cypress",
"Spring Boot",
"Python",
".NET",
"Contract testing",
"PostgreSQL",
],
dealBreakers: [],
preferredWorkArrangement: ["remote", "hybrid"],
preferredLocations: ["United States", "U.S.", "USA", "Remote US", "TN"],
minimumSalary: "",
industriesToTarget: [
"Software",
"FinTech",
"Financial services",
"Enterprise SaaS",
],
industriesToAvoid: [],
aboutMe:
"Ilia Dobkin (US) — Senior SDET / QA automation; Canadian citizen, TN-eligible (USMCA), open to U.S. relocation. Computer Systems Analyst-aligned framing. Switch to this profile in the Jobs page picker (same login as Canada).",
basicAuthUser: "ilia,dobkin",
resumeLocalPath: "../data/resumes/ilia-dobkin-us.json",
},
}),
`UPDATE search_profiles
SET data = json_set(data, '$.targetRoles', '${sqlJsonLiteral(ILIA_DOBKIN_CA_TARGET_ROLES)}'),
updated_at = datetime('now')
WHERE id = '685b0000-0000-4000-8000-000000000001'`,
`UPDATE keyword_sets
SET terms = '${sqlJsonLiteral(ILIA_DOBKIN_CA_KEYWORD_TERMS)}',
updated_at = datetime('now')
WHERE id = 'ilia-dobkin-keyword-set'`,
`INSERT OR IGNORE INTO keyword_sets (id, owner_profile_id, name, terms, is_active, created_at, updated_at)
VALUES (
'ilia-dobkin-us-keyword-set',
'685b0000-0000-4000-8000-000000000003',
'SDET',
'${sqlJsonLiteral(ILIA_DOBKIN_US_KEYWORD_TERMS)}',
1,
datetime('now'),
datetime('now')
)`,
`UPDATE search_profiles
SET data = json_set(
json_set(data, '$.basicAuthUser', 'ilia,dobkin'),
'$.aboutMe',
'Ilia Dobkin (US) — Senior SDET / QA automation; Canadian citizen, TN-eligible (USMCA), open to U.S. relocation. Computer Systems Analyst-aligned framing. Switch to this profile in the Jobs page picker (same login as Canada).'
),
updated_at = datetime('now')
WHERE id = '685b0000-0000-4000-8000-000000000003'`,
`UPDATE search_profiles
SET data = json_set(data, '$.targetRoles', '${sqlJsonLiteral(ILIA_DOBKIN_US_TARGET_ROLES)}'),
updated_at = datetime('now')
WHERE id = '685b0000-0000-4000-8000-000000000003'`,
`UPDATE keyword_sets
SET terms = '${sqlJsonLiteral(ILIA_DOBKIN_US_KEYWORD_TERMS)}',
updated_at = datetime('now')
WHERE id = 'ilia-dobkin-us-keyword-set'`,
// Remove duplicate Ilia profiles left by repository tests (same names, test UUIDs).
`DELETE FROM keyword_sets
WHERE owner_profile_id IN (
'685b0000-0000-4000-8000-000000009901',
'685b0000-0000-4000-8000-000000009902'
)`,
`DELETE FROM search_profiles
WHERE id IN (
'685b0000-0000-4000-8000-000000009901',
'685b0000-0000-4000-8000-000000009902'
)`,
];
console.log("🔧 Running database migrations...");

View File

@ -50,6 +50,9 @@ async function loadJobDedupIndexes(ownerProfileId: string): Promise<{
existingContentFingerprintSet: Set<string>;
dismissedDedupKeySet: Set<string>;
}> {
const dismissLookbackMs = 90 * 24 * 60 * 60 * 1000;
const dismissCutoffMs = Date.now() - dismissLookbackMs;
const rows = await db
.select({
jobUrl: jobs.jobUrl,
@ -60,6 +63,7 @@ async function loadJobDedupIndexes(ownerProfileId: string): Promise<{
title: jobs.title,
jobDescription: jobs.jobDescription,
status: jobs.status,
updatedAt: jobs.updatedAt,
})
.from(jobs)
.where(eq(jobs.ownerProfileId, ownerProfileId));
@ -94,12 +98,15 @@ async function loadJobDedupIndexes(ownerProfileId: string): Promise<{
}
if (row.status === "skipped" || row.status === "applied") {
for (const key of collectJobDedupKeys({
employer: row.employer,
title: row.title,
jobDescription: row.jobDescription,
})) {
dismissedDedupKeySet.add(key);
const updatedMs = Date.parse(row.updatedAt);
if (Number.isFinite(updatedMs) && updatedMs >= dismissCutoffMs) {
for (const key of collectJobDedupKeys({
employer: row.employer,
title: row.title,
jobDescription: row.jobDescription,
})) {
dismissedDedupKeySet.add(key);
}
}
}
}

View File

@ -0,0 +1,108 @@
import { eq, inArray } from "drizzle-orm";
import { afterEach, beforeEach, describe, expect, it } from "vitest";
import { db } from "../db";
import { searchProfiles } from "../db/schema";
import {
getSearchProfileIdForBasicAuthUser,
listProfilesForBasicAuthUser,
profileMatchesBasicAuthUser,
setActiveSearchProfileIdForBasicAuthUser,
} from "./profiles";
const CA_PROFILE_ID = "685b0000-0000-4000-8000-000000009901";
const US_PROFILE_ID = "685b0000-0000-4000-8000-000000009902";
const sharedProfileData = {
targetRoles: ["SDET"],
experienceLevel: "Senior",
mustHaveSkills: [],
niceToHaveSkills: [],
dealBreakers: [],
preferredWorkArrangement: [],
preferredLocations: [],
minimumSalary: "",
industriesToTarget: [],
industriesToAvoid: [],
aboutMe: "",
basicAuthUser: "__test_profiles_repo__",
};
describe("profiles repository", () => {
afterEach(async () => {
await db
.delete(searchProfiles)
.where(inArray(searchProfiles.id, [CA_PROFILE_ID, US_PROFILE_ID]));
});
beforeEach(async () => {
await db
.delete(searchProfiles)
.where(inArray(searchProfiles.id, [CA_PROFILE_ID, US_PROFILE_ID]));
await db.insert(searchProfiles).values([
{
id: CA_PROFILE_ID,
name: "Test Profiles Repo CA",
data: JSON.stringify({
...sharedProfileData,
preferredLocations: ["Canada"],
resumeLocalPath: "../data/resumes/ilia-dobkin.json",
}),
},
{
id: US_PROFILE_ID,
name: "Test Profiles Repo US",
data: JSON.stringify({
...sharedProfileData,
preferredLocations: ["United States"],
resumeLocalPath: "../data/resumes/ilia-dobkin-us.json",
}),
},
]);
});
it("matches comma-separated basic auth aliases", () => {
expect(profileMatchesBasicAuthUser("ilia,dobkin", "ILIA")).toBe(true);
expect(profileMatchesBasicAuthUser("ilia,dobkin", "dobkin")).toBe(true);
expect(profileMatchesBasicAuthUser("ilia,dobkin", "cherepaha")).toBe(false);
});
it("lists every profile visible to the login", async () => {
const visible = await listProfilesForBasicAuthUser(
"__test_profiles_repo__",
);
expect(visible.map((profile) => profile.id).sort()).toEqual(
[CA_PROFILE_ID, US_PROFILE_ID].sort(),
);
});
it("defaults to the first matching profile when none is stored", async () => {
await expect(
getSearchProfileIdForBasicAuthUser("__test_profiles_repo__"),
).resolves.toBe(CA_PROFILE_ID);
});
it("remembers the active profile across basic auth aliases", async () => {
await db
.update(searchProfiles)
.set({
data: JSON.stringify({
...sharedProfileData,
preferredLocations: ["Canada"],
basicAuthUser: "test-repo-a,test-repo-b",
}),
})
.where(eq(searchProfiles.id, US_PROFILE_ID));
await setActiveSearchProfileIdForBasicAuthUser(
"test-repo-a",
US_PROFILE_ID,
);
await expect(
getSearchProfileIdForBasicAuthUser("test-repo-a"),
).resolves.toBe(US_PROFILE_ID);
await expect(
getSearchProfileIdForBasicAuthUser("test-repo-b"),
).resolves.toBe(US_PROFILE_ID);
});
});

View File

@ -8,6 +8,13 @@ import type {
import { eq } from "drizzle-orm";
import { db } from "../db";
import { searchProfiles } from "../db/schema";
import type { SettingKey } from "./settings";
import { getSetting, setSetting } from "./settings";
const BASIC_AUTH_ACTIVE_PROFILE_BY_USER_KEY: SettingKey =
"basicAuthActiveProfileByUser";
type BasicAuthActiveProfileByUser = Record<string, string>;
function mapRow(row: typeof searchProfiles.$inferSelect): SearchProfile {
let data: JobSearchProfile;
@ -110,16 +117,76 @@ export async function updateProfile(
return getProfileById(id);
}
function parseBasicAuthActiveProfileByUser(
raw: string | null | undefined,
): BasicAuthActiveProfileByUser {
if (!raw?.trim()) return {};
try {
const parsed: unknown = JSON.parse(raw);
if (!parsed || typeof parsed !== "object" || Array.isArray(parsed)) {
return {};
}
const out: BasicAuthActiveProfileByUser = {};
for (const [key, value] of Object.entries(parsed)) {
const alias = key.trim().toLowerCase();
const profileId = typeof value === "string" ? value.trim() : "";
if (alias && profileId) out[alias] = profileId;
}
return out;
} catch {
return {};
}
}
async function loadBasicAuthActiveProfileByUser(): Promise<BasicAuthActiveProfileByUser> {
const raw = await getSetting(BASIC_AUTH_ACTIVE_PROFILE_BY_USER_KEY);
return parseBasicAuthActiveProfileByUser(raw);
}
function basicAuthAliases(basicAuthUser: string | null | undefined): string[] {
return (basicAuthUser ?? "")
.split(",")
.map((part) => part.trim().toLowerCase())
.filter(Boolean);
}
/**
* When one login maps to multiple search profiles, remember which profile is active.
* Writes the same profile id for every alias on that profile (e.g. ilia + dobkin).
*/
export async function setActiveSearchProfileIdForBasicAuthUser(
username: string,
profileId: string,
): Promise<void> {
const profile = await getProfileById(profileId);
if (!profile) return;
const requestAlias = username.trim().toLowerCase();
if (!requestAlias) return;
if (!profileMatchesBasicAuthUser(profile.data.basicAuthUser, requestAlias)) {
return;
}
const map = await loadBasicAuthActiveProfileByUser();
for (const alias of basicAuthAliases(profile.data.basicAuthUser)) {
map[alias] = profileId;
}
await setSetting(BASIC_AUTH_ACTIVE_PROFILE_BY_USER_KEY, JSON.stringify(map));
}
export async function getSearchProfileIdForBasicAuthUser(
username: string,
): Promise<string | null> {
const profiles = await listProfiles();
for (const p of profiles) {
if (profileMatchesBasicAuthUser(p.data.basicAuthUser, username)) {
return p.id;
}
const matches = await listProfilesForBasicAuthUser(username);
if (matches.length === 0) return null;
if (matches.length === 1) return matches[0].id;
const map = await loadBasicAuthActiveProfileByUser();
const preferred = map[username.trim().toLowerCase()];
if (preferred && matches.some((profile) => profile.id === preferred)) {
return preferred;
}
return null;
return matches[0].id;
}
export async function deleteProfile(id: string): Promise<boolean> {

View File

@ -151,19 +151,7 @@ export async function getEffectiveSettings(): Promise<AppSettings> {
...envSettings,
};
// In Basic Auth mode, the "active" search profile is derived from the request
// context (basic auth username → profile.basicAuthUser), not from the stored
// activeProfileId setting. Expose that derived id so the UI updates the right
// profile record on save.
const requestOwnerProfileId = getJobOwnerProfileId();
if (
tenantJobSearchProfile &&
requestOwnerProfileId &&
requestOwnerProfileId !== DEFAULT_JOB_OWNER_PROFILE_ID &&
requestOwnerProfileId !== "__unmapped__"
) {
result.activeProfileId = requestOwnerProfileId;
}
const rawModel = overrides.model;
const modelDef = settingsRegistry.model;
@ -254,6 +242,19 @@ export async function getEffectiveSettings(): Promise<AppSettings> {
}
}
// In Basic Auth mode, the active search profile is derived from the request
// context (username → basicAuthActiveProfileByUser), not the stored
// activeProfileId row. Apply after the string-settings loop, which would
// otherwise overwrite this field from the global settings table.
if (
tenantJobSearchProfile &&
requestOwnerProfileId &&
requestOwnerProfileId !== DEFAULT_JOB_OWNER_PROFILE_ID &&
requestOwnerProfileId !== "__unmapped__"
) {
result.activeProfileId = requestOwnerProfileId;
}
// Always expose the effective base resume id for the active RxResume mode.
result.rxresumeBaseResumeId = rxresumeBaseResumeId;

35
package-lock.json generated
View File

@ -336,6 +336,37 @@
"undici-types": "~7.16.0"
}
},
"extractors/google-jobs": {
"name": "google-jobs-extractor",
"version": "0.0.1",
"dependencies": {
"camoufox-js": "^0.8.0",
"job-ops-shared": "^1.0.0",
"playwright": "^1.57.0",
"tsx": "^4.4.0"
},
"devDependencies": {
"@types/node": "^24.0.0",
"typescript": "~5.9.0"
}
},
"extractors/google-jobs/node_modules/@types/node": {
"version": "24.13.3",
"resolved": "https://registry.npmjs.org/@types/node/-/node-24.13.3.tgz",
"integrity": "sha512-Dh8vAsV36ig5wa9OX4pXvMc9D3Veibfw2wix0CUwYODLD8nkj9UsLjASr49nPg+2eKzxhBV+v7L8pXvT4e639Q==",
"dev": true,
"license": "MIT",
"dependencies": {
"undici-types": "~7.18.0"
}
},
"extractors/google-jobs/node_modules/undici-types": {
"version": "7.18.2",
"resolved": "https://registry.npmjs.org/undici-types/-/undici-types-7.18.2.tgz",
"integrity": "sha512-AsuCzffGHJybSaRrmr5eHr81mwJU3kjw6M+uprWvCXiNeN9SOGwQ3Jn8jb8m3Z6izVgknn1R0FTCEAP2QrLY/w==",
"dev": true,
"license": "MIT"
},
"extractors/gradcracker": {
"name": "gradcracker-extractor",
"version": "0.0.1",
@ -14253,6 +14284,10 @@
"url": "https://github.com/sponsors/sindresorhus"
}
},
"node_modules/google-jobs-extractor": {
"resolved": "extractors/google-jobs",
"link": true
},
"node_modules/gopd": {
"version": "1.2.0",
"resolved": "https://registry.npmjs.org/gopd/-/gopd-1.2.0.tgz",

View File

@ -0,0 +1,30 @@
#
# Jobber cron env — U.S. pipeline (optional separate cron host)
# Copy to /root/.jobber-cron-dobkin-us.env (chmod 600)
#
# Note: the U.S. profile is also available in the UI under the same ilia/dobkin
# login — use the Jobs page profile picker to switch. This env file is only
# needed if you want a dedicated cron that always runs the U.S. profile without
# switching in the UI (set BASIC_AUTH_ACTIVE_PROFILE_BY_USER or activate once).
#
# Used by: scripts/jobber-pipeline-telegram.sh
#
# JobOps base URL (where the app is reachable from the cron host)
JOBOPS_URL=http://127.0.0.1:3005
# Optional: limit number of jobs linked in Telegram message
JOB_TELEGRAM_MAX_JOBS=25
# Optional: comma-separated sources to run (leave empty to use server defaults)
# JOBBER_PIPELINE_SOURCES=usajobs,adzuna,remoteok,remotive
# Same login as Canada; activate the U.S. profile once in Settings/Jobs picker,
# or set basicAuthActiveProfileByUser in the DB for this user.
BASIC_AUTH_USER=dobkin
BASIC_AUTH_PASSWORD=CHANGEME
# Telegram bot + chat destination
TELEGRAM_BOT_TOKEN=CHANGEME
TELEGRAM_CHAT_ID=CHANGEME

View File

@ -340,6 +340,14 @@ const ALL_TARGETS: Target[] = [
? undefined
: "Camoufox not installed (run: npx camoufox-js fetch)",
},
{
id: "google-jobs",
importPath: "../extractors/google-jobs/manifest",
settings: { googleJobsMaxJobsPerTerm: "5" },
skipReason: camoufoxInstalled()
? undefined
: "Camoufox not installed (run: npx camoufox-js fetch)",
},
{
id: "workday",
importPath: "../extractors/workday/manifest",

View File

@ -26,6 +26,7 @@ export const EXTRACTOR_SOURCE_IDS = [
"testdevjobs",
"wellfound",
"builtin",
"google-jobs",
// --- Public ATS / career-page sources ---
"ashby",
"lever",
@ -217,6 +218,12 @@ export const EXTRACTOR_SOURCE_METADATA: Record<
category: "pipeline",
region: "global",
},
"google-jobs": {
label: "Google Jobs",
order: 199,
category: "pipeline",
region: "global",
},
ashby: {
label: "Ashby (ATS)",
order: 210,

View File

@ -3,8 +3,11 @@ import {
buildJobContentFingerprint,
buildJobDescriptionFingerprint,
collectJobDedupKeys,
employersMatchForDismiss,
jobsMatchEmployerAndTitleDismiss,
normalizeEmployerForFingerprint,
normalizeTitleForFingerprint,
titlesMatchForDismiss,
} from "./job-fingerprint";
describe("buildJobContentFingerprint", () => {
@ -96,10 +99,58 @@ describe("buildJobContentFingerprint", () => {
expect(normalizeEmployerForFingerprint("Acme GmbH")).toBe("acme");
});
it("normalizeEmployerForFingerprint strips leading punctuation", () => {
expect(normalizeEmployerForFingerprint(". CGI IT UK Limited")).toBe(
"cgiituklimited",
);
});
it("normalizeTitleForFingerprint drops leading repost markers", () => {
expect(normalizeTitleForFingerprint("[Reposted] Software Engineer")).toBe(
"softwareengineer",
);
});
});
describe("prior skip dismiss matching", () => {
it("employersMatchForDismiss allows prefix matches", () => {
expect(
employersMatchForDismiss("cgi", normalizeEmployerForFingerprint("CGI")),
).toBe(true);
expect(
employersMatchForDismiss(
normalizeEmployerForFingerprint("CGI IT UK Limited"),
normalizeEmployerForFingerprint("CGI"),
),
).toBe(true);
});
it("titlesMatchForDismiss allows plural variants", () => {
expect(
titlesMatchForDismiss(
normalizeTitleForFingerprint("Automation Test Engineer"),
normalizeTitleForFingerprint("Automation Test Engineers"),
),
).toBe(true);
});
it("jobsMatchEmployerAndTitleDismiss requires same title family", () => {
expect(
jobsMatchEmployerAndTitleDismiss({
employerA: "CGI IT UK Limited",
titleA: "Automation Test Engineers",
employerB: "CGI",
titleB: "Automation Test Engineer",
}),
).toBe(true);
expect(
jobsMatchEmployerAndTitleDismiss({
employerA: "Cognizant",
titleA: "AI Automation Test Engineer",
employerB: "Cognizant",
titleB: "Automation Test Engineer",
}),
).toBe(false);
});
});
});

View File

@ -29,6 +29,9 @@ const TRAILING_CITY_REGION_RE = /\s+[-|–—]\s+[^,\-|–—]+,\s*[^,\-|–—]
const COMPANY_LEGAL_SUFFIX_RE =
/\b(?:inc|inc\.|ltd|ltd\.|llc|gmbh|s\.a\.|s\.r\.l|sa|nv|bv|plc|corp|corporation|co|company|holdings|holding)\b/g;
/** Hide rediscovered roles that match a prior skip/apply within this window. */
export const PRIOR_SKIP_DISMISS_LOOKBACK_MS = 90 * 24 * 60 * 60 * 1000;
function stripDiacritics(input: string): string {
return input.normalize("NFKD").replace(/[\u0300-\u036f]/g, "");
}
@ -38,6 +41,7 @@ export function normalizeEmployerForFingerprint(
): string {
if (!employer) return "";
let value = stripDiacritics(employer.toLowerCase()).trim();
value = value.replace(/^[^a-z0-9]+/g, "");
value = value.replace(PARENS_RE, " ");
value = value.replace(COMPANY_LEGAL_SUFFIX_RE, " ");
value = value.replace(PUNCTUATION_RE, " ");
@ -103,6 +107,69 @@ export function buildJobDescriptionFingerprint(args: {
return `${employer}::desc::${description}`;
}
/**
* Whether two normalized employers refer to the same company for dismiss matching.
* Allows prefix matches so `cgi` matches `cgiituklimited`.
*/
export function employersMatchForDismiss(
normalizedEmployerA: string,
normalizedEmployerB: string,
): boolean {
if (!normalizedEmployerA || !normalizedEmployerB) return false;
if (normalizedEmployerA === normalizedEmployerB) return true;
const [short, long] =
normalizedEmployerA.length <= normalizedEmployerB.length
? [normalizedEmployerA, normalizedEmployerB]
: [normalizedEmployerB, normalizedEmployerA];
if (short.length < 3) return false;
return long.startsWith(short);
}
/**
* Whether two normalized titles are the same role for dismiss matching.
* Allows a trailing plural `s` variant (`engineer` vs `engineers`).
*/
export function titlesMatchForDismiss(
normalizedTitleA: string,
normalizedTitleB: string,
): boolean {
if (!normalizedTitleA || !normalizedTitleB) return false;
if (normalizedTitleA === normalizedTitleB) return true;
if (
normalizedTitleA.length > 4 &&
normalizedTitleA.endsWith("s") &&
normalizedTitleA.slice(0, -1) === normalizedTitleB
) {
return true;
}
if (
normalizedTitleB.length > 4 &&
normalizedTitleB.endsWith("s") &&
normalizedTitleB.slice(0, -1) === normalizedTitleA
) {
return true;
}
return false;
}
/** Same company + same job title (normalized), for prior-skip hide rules. */
export function jobsMatchEmployerAndTitleDismiss(args: {
employerA: string | null | undefined;
titleA: string | null | undefined;
employerB: string | null | undefined;
titleB: string | null | undefined;
}): boolean {
const employerA = normalizeEmployerForFingerprint(args.employerA);
const employerB = normalizeEmployerForFingerprint(args.employerB);
const titleA = normalizeTitleForFingerprint(args.titleA);
const titleB = normalizeTitleForFingerprint(args.titleB);
if (!employerA || !employerB || !titleA || !titleB) return false;
return (
employersMatchForDismiss(employerA, employerB) &&
titlesMatchForDismiss(titleA, titleB)
);
}
export function collectJobDedupKeys(args: {
employer: string | null | undefined;
title: string | null | undefined;

View File

@ -480,6 +480,13 @@ export const settingsRegistry = {
parse: parseIntOrNull,
serialize: serializeNullableNumber,
},
googleJobsMaxJobsPerTerm: {
kind: "typed" as const,
schema: z.number().int().min(1).max(150),
default: (): number => 30,
parse: parseIntOrNull,
serialize: serializeNullableNumber,
},
fourdayweekMaxJobsPerTerm: {
kind: "typed" as const,
schema: z.number().int().min(1).max(1000),
@ -965,6 +972,10 @@ export const settingsRegistry = {
kind: "string" as const,
schema: z.string().trim().max(200),
},
basicAuthActiveProfileByUser: {
kind: "string" as const,
schema: z.string().trim().max(10000),
},
rxresumeBaseResumeId: {
kind: "string" as const,
schema: z.string().trim().max(200),

View File

@ -182,6 +182,7 @@ export const createAppSettings = (
override: null,
},
activeProfileId: null,
basicAuthActiveProfileByUser: null,
rxresumeBaseResumeId: null,
rxresumeBaseResumeIdV4: null,
rxresumeBaseResumeIdV5: null,
@ -206,6 +207,7 @@ export const createAppSettings = (
builtinMaxJobsPerTerm: { value: 100, default: 100, override: null },
builtinMaxPagesPerTerm: { value: 3, default: 3, override: null },
wellfoundMaxJobsPerTerm: { value: 50, default: 50, override: null },
googleJobsMaxJobsPerTerm: { value: 30, default: 30, override: null },
fourdayweekMaxJobsPerTerm: { value: 50, default: 50, override: null },
qajobsboardMaxJobsPerTerm: { value: 100, default: 100, override: null },
arcRemoteJobsPaths: {

View File

@ -218,6 +218,7 @@ export interface AppSettings {
builtinMaxJobsPerTerm: Resolved<number>;
builtinMaxPagesPerTerm: Resolved<number>;
wellfoundMaxJobsPerTerm: Resolved<number>;
googleJobsMaxJobsPerTerm: Resolved<number>;
fourdayweekMaxJobsPerTerm: Resolved<number>;
qajobsboardMaxJobsPerTerm: Resolved<number>;
arcRemoteJobsPaths: Resolved<string[]>;
@ -270,6 +271,8 @@ export interface AppSettings {
// Simple strings:
activeProfileId: string | null;
/** JSON map of basic-auth username → search profile id when one login has multiple profiles. */
basicAuthActiveProfileByUser: string | null;
rxresumeBaseResumeId: string | null;
rxresumeBaseResumeIdV4: string | null;
rxresumeBaseResumeIdV5: string | null;