# UKVisaJobs Extractor (How It Works)

This is a plain-English walkthrough of the UK Visa Jobs extractor. It's the most complex one because the site requires an authenticated session before the API will return jobs.

## Big picture

There are two layers:

1) `extractors/ukvisajobs/src/main.ts` handles logging in, talking to the UKVisaJobs API, and writing a Crawlee-style dataset.
2) `orchestrator/src/server/services/ukvisajobs.ts` runs that extractor, reads the dataset, de-dupes results, and optionally enriches descriptions.

## 1) Authentication and session cache

The API requires a token + cookies. The extractor keeps these in a cache file:

- `extractors/ukvisajobs/storage/ukvisajobs-auth.json`

Flow:

- If there's a cached session, it uses it.
- If not, it launches a real browser (Playwright + Camoufox), logs in with `UKVISAJOBS_EMAIL` and `UKVISAJOBS_PASSWORD`, then captures the auth cookies + token.
- It stores those values in the cache file for reuse.

You can force a refresh with:

- `UKVISAJOBS_REFRESH_ONLY=1`

If the API responds with an "expired" token error, it will automatically re-login and retry.

## 2) API requests

Once authenticated, it posts to:

- `https://my.ukvisajobs.com/ukvisa-api/api/fetch-jobs-data`

Each request:

- Includes the auth token in a form field.
- Includes cookies in the header (`csrf_token`, `ci_session`, `authToken`).
- Filters by search keyword if provided.
- Uses pagination (15 jobs per page).

## 3) Job mapping

The extractor normalizes the raw API data into the project's job shape:

- Salary is built from min/max values and interval.
- Visa-related flags are turned into a short fallback description if the job has no real description.
- The `job_link` becomes both `jobUrl` and `applicationLink`.

## 4) Output dataset

The extractor writes the results to:

- `extractors/ukvisajobs/storage/datasets/default/`

It mirrors Crawlee's dataset format:

- One JSON file per job.
- A combined `jobs.json` containing all jobs.

## 5) Orchestrator flow (how the app uses it)

When the pipeline runs:

- The server spawns the extractor as a child process (`npx tsx src/main.ts`).
- It can run multiple search terms sequentially (with a short delay between them).
- It reads the dataset and de-dupes by `sourceJobId` (or `jobUrl` fallback).
- If a job's description is missing or too short, it makes a direct HTTP request to the job URL and extracts plain text.
  - This is effectively a curl-style fetch of the job page to fill in the JD for scoring and summarization.

## Controls and limits

Key environment variables:

- `UKVISAJOBS_EMAIL`, `UKVISAJOBS_PASSWORD` (required for auth refresh)
- `UKVISAJOBS_HEADLESS` (set `false` to show the browser)
- `UKVISAJOBS_MAX_JOBS` (default 50, max 200)
- `UKVISAJOBS_SEARCH_KEYWORD` (single keyword filter)

The UI also lets you set max jobs and search terms via the pipeline settings.

## Practical notes

- If you remove the auth cache file, the next run will re-login.
- The extractor is intentionally polite: it runs low concurrency and adds short delays.
- If the API or session changes on the UKVisaJobs side, the refresh logic is the first thing to check.