Jobber/docs-site/docs/workflows/add-an-extractor.md

id, title, description, sidebar_position

id	title	description	sidebar_position
add-an-extractor	Add an Extractor	How to add a new extractor using the manifest contract and shared extractor catalog.	2

What it is

This guide explains how to add a new extractor that is auto-registered at orchestrator startup.

The extractor runtime is discovered from a local manifest.ts file, and the source is type-safe across API/client through the shared catalog in shared/src/extractors/index.ts.

Extractor manifests must live in extractor packages under extractors/<name>/ only. Do not add manifest files inside orchestrator/. Extractor run logic should also live in the extractor package so orchestrator stays extractor-agnostic.

Why it exists

Without a manifest contract, adding extractors required touching multiple orchestrator files.

With the manifest system, contributors only need to:

1. Add a manifest in their extractor package.
2. Add the new source id to the shared typed catalog.

That keeps runtime wiring dynamic while preserving compile-time safety in API and client code.

How to use it

1. Create your extractor package under extractors/<name>/.
2. Add a manifest.ts in the extractor package root (or src/manifest.ts).
   • Valid locations are only extractors/<name>/manifest.ts or extractors/<name>/src/manifest.ts.
   • orchestrator/**/manifest.ts is not used for extractor discovery.
3. Export a manifest with:
   • id
   • displayName
   • providesSources
   • requiredEnvVars (optional)
   • run(context) that returns { success, jobs, error? }
4. Add the new source id to shared/src/extractors/index.ts:
   • append to EXTRACTOR_SOURCE_IDS
   • add an entry in EXTRACTOR_SOURCE_METADATA
5. Ensure your extractor maps output to CreateJobInput[].
6. Register it in scripts/smoke-extractors.ts (ALL_TARGETS): add one row per manifest so npx tsx scripts/smoke-extractors.ts exercises every shipped extractor (keyed sources SKIP until env vars exist).
7. Run the full CI checks.

Example manifest:

import type { ExtractorManifest } from "@shared/types/extractors";

export const manifest: ExtractorManifest = {
  id: "myextractor",
  displayName: "My Extractor",
  providesSources: ["myextractor"],
  requiredEnvVars: ["MYEXTRACTOR_API_KEY"],
  async run(context) {
    // context.searchTerms, context.settings, context.onProgress, context.shouldCancel
    const jobs = [];
    return { success: true, jobs };
  },
};

export default manifest;

Subprocess extractors are supported. Keep subprocess spawning inside run(context) so orchestrator only depends on the manifest contract.

Common problems

Extractor not discovered at startup

• Check file path: extractors/<name>/manifest.ts or extractors/<name>/src/manifest.ts.
• Ensure the file exports default or named manifest.

Source compiles in extractor but fails in API/client

• Add the new source id to shared/src/extractors/index.ts.
• Confirm metadata exists for that source id.

Smoke connectivity

After wiring settings/env, run:

npm run smoke:extractors -- myextractor

Or the full suite (may take several minutes — JobSpy invokes Python; Gradcracker / Hiring Cafe need Camoufox: npx camoufox-js fetch):

npm run smoke:extractors

Keep ALL_TARGETS in that script aligned with manifests under each extractors/<name>/ package (manifest.ts or src/manifest.ts).

Source appears in shared catalog but is unavailable at runtime

• The manifest was not loaded successfully.
• Check startup logs for registry warnings.

Source requires credentials but never returns jobs

• Add and validate requiredEnvVars.
• Verify your manifest run(context) reads settings/env values correctly.

Related pages

• Extractors Overview
• Adzuna Extractor
• Hiring Cafe Extractor
• UKVisaJobs Extractor