Add startup.jobs extractor support (#279)
* Add startup.jobs extractor support * Harden startup.jobs extractor inputs * Wire startupjobs into Docker and CI * Tighten startupjobs review follow-ups * fix: publish ghcr during release workflow * feat: add startupjobs max jobs configuration and update related tests
This commit is contained in:
Shaheer Sarfaraz
@@ -17,6 +17,7 @@ Extractor integrations are now registered through manifests and loaded automatic
|
||||
| [JobSpy](/docs/next/extractors/jobspy) | Multi-source discovery (Indeed, LinkedIn, Glassdoor) | Requires Python wrapper execution per term; source availability and quality vary by site/location | `JOBSPY_SITES`, `JOBSPY_SEARCH_TERMS`, `JOBSPY_RESULTS_WANTED`, `JOBSPY_HOURS_OLD`, `JOBSPY_LINKEDIN_FETCH_DESCRIPTION` | Produces JSON per term, then orchestrator normalizes and de-duplicates by `jobUrl` |
|
||||
| [Adzuna](/docs/next/extractors/adzuna) | API-based multi-country discovery with low scraping overhead | Requires valid App ID/App Key; country must be in Adzuna-supported list | `ADZUNA_APP_ID`, `ADZUNA_APP_KEY`, `ADZUNA_MAX_JOBS_PER_TERM` | API pagination to dataset output; orchestrator maps progress and de-duplicates by `sourceJobId`/`jobUrl` |
|
||||
| [Hiring Cafe](/docs/next/extractors/hiring-cafe) | Browser-backed discovery using Hiring Cafe search APIs | Subject to upstream anti-bot checks; uses browser context and encoded search-state payloads | `HIRING_CAFE_SEARCH_TERMS`, `HIRING_CAFE_COUNTRY`, `HIRING_CAFE_MAX_JOBS_PER_TERM`, `HIRING_CAFE_DATE_FETCHED_PAST_N_DAYS` | Uses existing pipeline term/country/budget knobs and maps directly to normalized jobs |
|
||||
| [startup.jobs](/docs/next/extractors/startup-jobs) | Startup-focused discovery through the published `startup-jobs-scraper` package | No credentials required; detail enrichment depends on Playwright browser binaries being installed | existing pipeline `searchTerms`, selected country/cities, `jobspyResultsWanted`; `npx playwright install` for fresh environments | Algolia-backed search plus detail-page enrichment via package import; orchestrator maps normalized records and de-duplicates by `jobUrl` |
|
||||
| [UKVisaJobs](/docs/next/extractors/ukvisajobs) | UK visa sponsorship-focused roles | Requires authenticated session and periodic token/cookie refresh | `UKVISAJOBS_EMAIL`, `UKVISAJOBS_PASSWORD`, `UKVISAJOBS_MAX_JOBS`, `UKVISAJOBS_SEARCH_KEYWORD` | API pagination + dataset output; orchestrator de-dupes and may fetch missing descriptions |
|
||||
| [Manual Import](/docs/next/extractors/manual) | One-off jobs not covered by scrapers | Inference quality depends on model/provider and input quality; some URLs cannot be fetched reliably | App/API endpoints (`/api/manual-jobs/infer`, `/api/manual-jobs/import`) | Accepts text/HTML/URL, runs inference, then saves and scores job after review |
|
||||
|
||||
@@ -25,6 +26,7 @@ Extractor integrations are now registered through manifests and loaded automatic
|
||||
- Use **JobSpy** for broad first-pass sourcing across common boards.
|
||||
- Use **Adzuna** when you want API-first discovery in supported non-UK markets.
|
||||
- Use **Hiring Cafe** when you want another term/country-driven source without adding credentials.
|
||||
- Use **startup.jobs** when you want startup-heavy listings without maintaining another scraper locally.
|
||||
- Use **Gradcracker** when targeting graduate pipelines in the UK.
|
||||
- Use **UKVisaJobs** for sponsorship-specific UK searches.
|
||||
- Use **Manual Import** when you already have a specific posting and need direct import.
|
||||
@@ -37,6 +39,7 @@ Many runs combine sources: broad discovery first, then manual import for high-pr
|
||||
- [JobSpy](/docs/next/extractors/jobspy)
|
||||
- [Adzuna](/docs/next/extractors/adzuna)
|
||||
- [Hiring Cafe](/docs/next/extractors/hiring-cafe)
|
||||
- [startup.jobs](/docs/next/extractors/startup-jobs)
|
||||
- [UKVisaJobs](/docs/next/extractors/ukvisajobs)
|
||||
- [Manual Import](/docs/next/extractors/manual)
|
||||
- [Add an Extractor](/docs/next/workflows/add-an-extractor)
|
||||
|
||||
@@ -0,0 +1,64 @@
|
||||
---
|
||||
id: startup-jobs
|
||||
title: startup.jobs Extractor
|
||||
description: startup.jobs extraction integrated through the startup-jobs-scraper package.
|
||||
sidebar_position: 8
|
||||
---
|
||||
|
||||
## What it is
|
||||
|
||||
Original website: [startup.jobs](https://startup.jobs)
|
||||
|
||||
This extractor wraps the published [`startup-jobs-scraper`](https://www.npmjs.com/package/startup-jobs-scraper) package and feeds normalized startup.jobs listings into the existing pipeline.
|
||||
|
||||
Implementation split:
|
||||
|
||||
1. `extractors/startupjobs/src/run.ts` calls `scrapeStartupJobsViaAlgolia` and maps package records into `CreateJobInput`.
|
||||
2. `extractors/startupjobs/src/manifest.ts` adapts pipeline settings, emits progress updates, and registers the source for runtime discovery.
|
||||
|
||||
## Why it exists
|
||||
|
||||
startup.jobs adds a startup-focused board to job-ops without introducing another bespoke scraper in this repository.
|
||||
|
||||
Using the published package also keeps the integration small and makes it easier to evolve the scraping logic independently from the app.
|
||||
|
||||
## How to use it
|
||||
|
||||
1. Open **Run jobs** and choose **Automatic**.
|
||||
2. Leave **startup.jobs** enabled in **Sources** or toggle it on.
|
||||
3. Set your usual automatic run controls:
|
||||
- `searchTerms` are sent as `query`.
|
||||
- country or city filters are reused as the package `location` option.
|
||||
- run budget path (`jobspyResultsWanted`) is reused as `requestedCount` per term.
|
||||
4. Start the run and monitor progress in the pipeline progress card.
|
||||
|
||||
Defaults and constraints:
|
||||
|
||||
- No new credentials are required.
|
||||
- The integration runs with `enrichDetails: true`, so it opens job detail pages for richer records.
|
||||
- Browser binaries are not downloaded automatically with the package. Install them with `npx playwright install` before using this extractor in a fresh environment.
|
||||
- When **Search cities** is set, the extractor runs once per city and once per search term.
|
||||
- Without explicit cities, the selected country is used as the location filter except for broad modes such as `worldwide` and `usa/ca`.
|
||||
|
||||
## Common problems
|
||||
|
||||
### startup.jobs does not appear in sources
|
||||
|
||||
- Check that the app is running a build that includes the new extractor manifest.
|
||||
- This source does not require credentials, so it should appear as soon as the updated build is loaded.
|
||||
|
||||
### Results are broader than expected
|
||||
|
||||
- If no city is configured, the extractor uses the selected country when possible and otherwise falls back to a broad search.
|
||||
- Add **Search cities** when you want tighter geographic filtering.
|
||||
|
||||
### Job descriptions are missing
|
||||
|
||||
- Detail enrichment depends on Playwright browser binaries being installed locally.
|
||||
- Run `npx playwright install` and retry if the extractor cannot open job detail pages.
|
||||
|
||||
## Related pages
|
||||
|
||||
- [Extractors Overview](/docs/next/extractors/overview)
|
||||
- [Pipeline Run](/docs/next/features/pipeline-run)
|
||||
- [Add an Extractor](/docs/next/workflows/add-an-extractor)
|
||||
@@ -49,6 +49,7 @@ const sidebars: SidebarsConfig = {
|
||||
"extractors/jobspy",
|
||||
"extractors/adzuna",
|
||||
"extractors/hiring-cafe",
|
||||
"extractors/startup-jobs",
|
||||
"extractors/manual",
|
||||
"extractors/ukvisajobs",
|
||||
],
|
||||
|
||||
Reference in New Issue
Block a user