feat(extractors): add 17 job source extractors and cross-source dedup
CI / Linting (Biome) (push) Failing after 36s
CI / Tests (push) Successful in 5m54s
CI / Type Check (adzuna-extractor) (push) Successful in 1m6s
CI / Type Check (gradcracker-extractor) (push) Successful in 1m9s
CI / Type Check (hiringcafe-extractor) (push) Successful in 1m5s
CI / Type Check (orchestrator) (push) Successful in 1m21s
CI / Type Check (startupjobs-extractor) (push) Successful in 1m4s
CI / Type Check (ukvisajobs-extractor) (push) Successful in 1m4s
CI / Documentation (push) Successful in 1m52s
CI / Linting (Biome) (push) Failing after 36s
CI / Tests (push) Successful in 5m54s
CI / Type Check (adzuna-extractor) (push) Successful in 1m6s
CI / Type Check (gradcracker-extractor) (push) Successful in 1m9s
CI / Type Check (hiringcafe-extractor) (push) Successful in 1m5s
CI / Type Check (orchestrator) (push) Successful in 1m21s
CI / Type Check (startupjobs-extractor) (push) Successful in 1m4s
CI / Type Check (ukvisajobs-extractor) (push) Successful in 1m4s
CI / Documentation (push) Successful in 1m52s
Adds extractor packages: arbeitnow, ashby, careerjet, fourdayweek,
greenhouse, himalayas, jobicy, jooble, lever, reed, remoteok, remotive,
themuse, usajobs, weworkremotely, workday — each with manifest, package
metadata and README.
Pipeline / shared:
- shared/job-fingerprint: stable hash for cross-source dedup, with tests
- discover-jobs: dedup via fingerprint and richer per-source merging
- jobs repository: fingerprint-aware upsert / lookup
- settings-registry, settings types/routes, demo-defaults: knobs for the
new sources
- shared extractors index: register the new manifests
- location-support, profiles route: small fixes for the new sources
Tooling:
- scripts/smoke-extractors.ts to sanity-check each source locally
- scripts/jobber-cron-{cherepaha,dobkin}.env.example: per-host cron
templates (CHANGEME placeholders only)
- .env.example: documented env vars for the new extractors
- .gitignore: ignore extractors/*/storage/ runtime caches (was ukvisajobs only)
Co-authored-by: Cursor <cursoragent@cursor.com>
This commit is contained in:
+113
@@ -97,3 +97,116 @@ ADZUNA_APP_KEY=
|
||||
# =============================================================================
|
||||
# Filter for remote-only jobs (default: 0 = disabled)
|
||||
# JOBSPY_IS_REMOTE=0
|
||||
|
||||
# =============================================================================
|
||||
# USAJOBS API (US federal jobs) - optional, US-only
|
||||
# =============================================================================
|
||||
# Register at https://developer.usajobs.gov/APIRequest/Index
|
||||
# USAJOBS requires a User-Agent that is a real contact email (per their TOS).
|
||||
# Leave unset to disable the source.
|
||||
# USAJOBS_API_KEY=
|
||||
# USAJOBS_USER_AGENT=you@example.com
|
||||
# USAJOBS_MAX_JOBS_PER_TERM=100
|
||||
|
||||
# =============================================================================
|
||||
# Jobicy (remote jobs feed) - optional, no auth
|
||||
# =============================================================================
|
||||
# Public JSON endpoint, capped at 50 results per call.
|
||||
# JOBICY_MAX_JOBS_PER_TERM=100
|
||||
|
||||
# =============================================================================
|
||||
# The Muse (jobs API) - optional, API key recommended
|
||||
# =============================================================================
|
||||
# https://www.themuse.com/developers/api/v2 — works without a key but is
|
||||
# heavily rate-limited. Set THEMUSE_API_KEY for higher quotas.
|
||||
# THEMUSE_API_KEY=
|
||||
# THEMUSE_MAX_JOBS_PER_TERM=100
|
||||
|
||||
# =============================================================================
|
||||
# Jooble (aggregator API) - optional
|
||||
# =============================================================================
|
||||
# Sign up at https://jooble.org/api/about for an API key.
|
||||
# JOOBLE_API_KEY=
|
||||
# JOOBLE_MAX_JOBS_PER_TERM=100
|
||||
|
||||
# =============================================================================
|
||||
# Careerjet (publisher API v4) - optional
|
||||
# =============================================================================
|
||||
# Register at https://www.careerjet.com/partners/api/ — declare API key + server IP(s).
|
||||
# CAREERJET_AFFID=your_api_key
|
||||
# CAREERJET_REFERER=https://your-site.com/path-to-job-search/
|
||||
# CAREERJET_USER_IP=203.0.113.1
|
||||
# Optional override for the required user_agent query param:
|
||||
# CAREERJET_USER_AGENT=Mozilla/5.0 ...
|
||||
# CAREERJET_MAX_JOBS_PER_TERM=100
|
||||
|
||||
# =============================================================================
|
||||
# Reed.co.uk (UK jobs API) - optional, UK-only
|
||||
# =============================================================================
|
||||
# Register at https://www.reed.co.uk/developers/jobseeker for an API key.
|
||||
# REED_API_KEY=
|
||||
# REED_MAX_JOBS_PER_TERM=100
|
||||
|
||||
# =============================================================================
|
||||
# Remote OK (remote jobs feed) - optional, no auth
|
||||
# =============================================================================
|
||||
# Public single-shot JSON feed at https://remoteok.com/api. We filter
|
||||
# client-side by your search terms (matched against position + tags).
|
||||
# Per Remote OK's TOS, link back to the original posting URLs when republishing.
|
||||
# REMOTEOK_MAX_JOBS_PER_TERM=100
|
||||
|
||||
# =============================================================================
|
||||
# Remotive (remote jobs feed) - optional, no auth
|
||||
# =============================================================================
|
||||
# Public JSON API at https://remotive.com/api/remote-jobs?limit=N&search=term.
|
||||
# Each search term is sent as the `search` parameter.
|
||||
# REMOTIVE_MAX_JOBS_PER_TERM=100
|
||||
|
||||
# =============================================================================
|
||||
# Arbeitnow (multi-ATS aggregator) - optional, no auth
|
||||
# =============================================================================
|
||||
# Public JSON API at https://www.arbeitnow.com/api/job-board-api?page=N.
|
||||
# Aggregates from Greenhouse, SmartRecruiters, Join, TeamTailor, Recruitee,
|
||||
# and Comeet. No server-side search; filtering is done client-side.
|
||||
# ARBEITNOW_MAX_JOBS_PER_TERM=100
|
||||
|
||||
# =============================================================================
|
||||
# Himalayas (remote jobs feed) - optional, no auth
|
||||
# =============================================================================
|
||||
# Public JSON API at https://himalayas.app/jobs/api?limit=N&offset=M.
|
||||
# No server-side search; filtering is done client-side by title + categories.
|
||||
# HIMALAYAS_MAX_JOBS_PER_TERM=100
|
||||
|
||||
# =============================================================================
|
||||
# We Work Remotely (RSS feed) - optional, no auth
|
||||
# =============================================================================
|
||||
# Public RSS at https://weworkremotely.com/remote-jobs.rss (all categories).
|
||||
# Single fetch; filtering is done client-side by title + skills + category.
|
||||
# WEWORKREMOTELY_MAX_JOBS_PER_TERM=100
|
||||
|
||||
# =============================================================================
|
||||
# 4 Day Week (reduced-schedule jobs) - optional, no auth
|
||||
# =============================================================================
|
||||
# Public JSON API at https://4dayweek.io/api/jobs?page=N.
|
||||
# Paginated; filtering is done client-side by title + tech stack.
|
||||
# No job description in listings; links to 4dayweek.io for details.
|
||||
# FOURDAYWEEK_MAX_JOBS_PER_TERM=100
|
||||
|
||||
# =============================================================================
|
||||
# Public ATS sources (Lever / Ashby / Greenhouse) - optional
|
||||
# =============================================================================
|
||||
# Comma- or newline-separated company slugs. The slug is the path segment used
|
||||
# in each provider's public job board, e.g. `lever.co/some-company` → "some-company".
|
||||
# LEVER_COMPANIES=netflix,figma
|
||||
# ASHBY_COMPANIES=ramp,linear
|
||||
# GREENHOUSE_COMPANIES=stripe,airbnb
|
||||
|
||||
# =============================================================================
|
||||
# Workday (public career sites) - optional
|
||||
# =============================================================================
|
||||
# Newline- or comma-separated entries. Each entry is either:
|
||||
# 1) A career-site URL we'll auto-parse, e.g.
|
||||
# https://nvidia.wd5.myworkdayjobs.com/en-US/NVIDIAExternalCareerSite
|
||||
# 2) A JSON object with explicit fields:
|
||||
# {"company":"NVIDIA","tenantUrl":"https://nvidia.wd5.myworkdayjobs.com","tenant":"nvidia","site":"NVIDIAExternalCareerSite","locale":"en-US"}
|
||||
# WORKDAY_TENANTS=
|
||||
|
||||
Reference in New Issue
Block a user