feat: keyword sets, sponsorship signals, and extractor/profile fixes
Add per-profile keyword sets, job source settings, sponsorship signal pills, and several ATS extractors. Fix Hiring Cafe discovery via Next.js SSR search, profile activate for comma-separated basicAuthUser aliases, and resume path backfill migrations. Update settings and hiring-cafe docs; localhost compose overlay for loopback-only deploys.
This commit is contained in:
@@ -11,11 +11,11 @@ Original website: [hiring.cafe](https://hiring.cafe)
|
||||
|
||||
Special thanks: Initial implementation inspiration came from [umur957/hiring-cafe-job-scraper](https://github.com/umur957/hiring-cafe-job-scraper).
|
||||
|
||||
Hiring Cafe is a browser-backed extractor that queries Hiring Cafe search APIs and maps results into the orchestrator `CreateJobInput` shape.
|
||||
Hiring Cafe is a browser-backed extractor that loads the site in Firefox (Camoufox when available), then fetches paginated search results through Hiring Cafe's Next.js SSR data endpoint and maps rows into the orchestrator `CreateJobInput` shape.
|
||||
|
||||
Implementation split:
|
||||
|
||||
1. `extractors/hiringcafe/src/main.ts` builds search state, calls Hiring Cafe APIs, and writes dataset JSON.
|
||||
1. `extractors/hiringcafe/src/main.ts` builds search state, reads the page `buildId`, calls `/_next/data/{buildId}/index.json?searchState=...&page=...`, and writes dataset JSON.
|
||||
2. `orchestrator/src/server/services/hiring-cafe.ts` runs the extractor, streams progress events, and maps rows for pipeline import.
|
||||
|
||||
## Why it exists
|
||||
@@ -57,6 +57,12 @@ npm --workspace hiringcafe-extractor run start
|
||||
|
||||
## Common problems
|
||||
|
||||
### Hiring Cafe returns 401 Unauthorized or 404 on `/api/search-jobs`
|
||||
|
||||
- Hiring Cafe removed or locked down the legacy GET `/api/search-jobs` and `/api/search-jobs/get-total-count` endpoints.
|
||||
- Current JobOps builds use the Next.js SSR transport (`ssrHits`, `ssrTotalCount`, `ssrIsLastPage`) instead.
|
||||
- Upgrade to a build that includes this change if logs still show `401` / `404` on the old API paths.
|
||||
|
||||
### Hiring Cafe returns 429 / Vercel security checkpoint
|
||||
|
||||
- The extractor first attempts Camoufox-backed Firefox and falls back to vanilla Firefox startup if Camoufox is unstable locally.
|
||||
|
||||
@@ -16,7 +16,7 @@ Extractor integrations are now registered through manifests and loaded automatic
|
||||
| [Gradcracker](/docs/next/extractors/gradcracker) | UK graduate roles from Gradcracker | Crawling stability depends on page structure and anti-bot behavior; tuned for low concurrency | `GRADCRACKER_SEARCH_TERMS`, `GRADCRACKER_MAX_JOBS_PER_TERM`, `JOBOPS_SKIP_APPLY_FOR_EXISTING` | Scrapes listing metadata, then detail pages and apply URL resolution |
|
||||
| [JobSpy](/docs/next/extractors/jobspy) | Multi-source discovery (Indeed, LinkedIn, Glassdoor) | Requires Python wrapper execution per term; source availability and quality vary by site/location | `JOBSPY_SITES`, `JOBSPY_SEARCH_TERMS`, `JOBSPY_RESULTS_WANTED`, `JOBSPY_HOURS_OLD`, `JOBSPY_LINKEDIN_FETCH_DESCRIPTION` | Produces JSON per term, then orchestrator normalizes and de-duplicates by `jobUrl` |
|
||||
| [Adzuna](/docs/next/extractors/adzuna) | API-based multi-country discovery with low scraping overhead | Requires valid App ID/App Key; country must be in Adzuna-supported list | `ADZUNA_APP_ID`, `ADZUNA_APP_KEY`, `ADZUNA_MAX_JOBS_PER_TERM` | API pagination to dataset output; orchestrator maps progress and de-duplicates by `sourceJobId`/`jobUrl` |
|
||||
| [Hiring Cafe](/docs/next/extractors/hiring-cafe) | Browser-backed discovery using Hiring Cafe search APIs | Subject to upstream anti-bot checks; uses browser context and encoded search-state payloads | `HIRING_CAFE_SEARCH_TERMS`, `HIRING_CAFE_COUNTRY`, `HIRING_CAFE_MAX_JOBS_PER_TERM`, `HIRING_CAFE_DATE_FETCHED_PAST_N_DAYS` | Uses existing pipeline term/country/budget knobs and maps directly to normalized jobs |
|
||||
| [Hiring Cafe](/docs/next/extractors/hiring-cafe) | Browser-backed discovery via Hiring Cafe Next.js SSR search payload | Subject to upstream anti-bot checks; uses browser context and JSON `searchState` query params | `HIRING_CAFE_SEARCH_TERMS`, `HIRING_CAFE_COUNTRY`, `HIRING_CAFE_MAX_JOBS_PER_TERM`, `HIRING_CAFE_DATE_FETCHED_PAST_N_DAYS` | Uses existing pipeline term/country/budget knobs and maps directly to normalized jobs |
|
||||
| [startup.jobs](/docs/next/extractors/startup-jobs) | Startup-focused discovery through the published `startup-jobs-scraper` package | No credentials required; detail enrichment depends on Playwright browser binaries being installed | existing pipeline `searchTerms`, selected country/cities, `jobspyResultsWanted`; `npx playwright install` for fresh environments | Algolia-backed search plus detail-page enrichment via package import; orchestrator maps normalized records and de-duplicates by `jobUrl` |
|
||||
| [UKVisaJobs](/docs/next/extractors/ukvisajobs) | UK visa sponsorship-focused roles | Requires authenticated session and periodic token/cookie refresh | `UKVISAJOBS_EMAIL`, `UKVISAJOBS_PASSWORD`, `UKVISAJOBS_MAX_JOBS`, `UKVISAJOBS_SEARCH_KEYWORD` | API pagination + dataset output; orchestrator de-dupes and may fetch missing descriptions |
|
||||
| [SmartRecruiters](/docs/next/extractors/smartrecruiters) | Enterprise employers on SmartRecruiters public boards | No auth; needs configured company identifiers; one HTTP round-trip per posting for apply URLs + descriptions | `SMARTRECRUITERS_COMPANIES`, `SMARTRECRUITERS_MAX_JOBS_PER_COMPANY` | Paginates the public Posting API, filters by pipeline terms, normalizes to `CreateJobInput` |
|
||||
|
||||
Reference in New Issue
Block a user