ilia
40c7cdece3
Ship a Camoufox-backed Google Jobs source, restore Glassdoor results discarded by a python-jobspy GraphQL quirk, and fix Himalayas/Gradcracker zero-job failures. Also harden prior-skip dismiss matching and multi-profile basic-auth switching for CA/US runs.
57 lines
2.0 KiB
Markdown
57 lines
2.0 KiB
Markdown
---
|
|
id: gradcracker
|
|
title: Gradcracker Extractor
|
|
description: How the Gradcracker crawler builds search URLs and extracts jobs.
|
|
sidebar_position: 2
|
|
---
|
|
|
|
A plain-English walkthrough of the Gradcracker extractor in `extractors/gradcracker`.
|
|
|
|
Original website: [gradcracker.com](https://www.gradcracker.com)
|
|
|
|
## Big picture
|
|
|
|
The crawler builds search URLs, scrapes listing pages, then opens job details for descriptions and apply URLs.
|
|
|
|
## 1) Build search URLs
|
|
|
|
- Combines UK regions with role terms.
|
|
- Defaults include roles such as `web-development` and `software-systems`.
|
|
- `GRADCRACKER_SEARCH_TERMS` overrides defaults.
|
|
|
|
## 2) Crawl list pages
|
|
|
|
- Waits for job cards (`article[wire:key]`).
|
|
- Extracts title, employer, discipline, deadline, salary, location, degree, start date.
|
|
- Queues job detail pages.
|
|
|
|
Controls:
|
|
|
|
- `GRADCRACKER_MAX_JOBS_PER_TERM`
|
|
- `JOBOPS_SKIP_APPLY_FOR_EXISTING=1`
|
|
- `JOBOPS_EXISTING_JOB_URLS` / `JOBOPS_EXISTING_JOB_URLS_FILE`
|
|
|
|
## 3) Crawl detail pages
|
|
|
|
- Waits for `.body-content`
|
|
- Captures full description text
|
|
- Clicks apply button to resolve final application URL
|
|
- Handles popup and same-tab redirects
|
|
|
|
## 4) Progress reporting
|
|
|
|
Set `JOBOPS_EMIT_PROGRESS=1` for structured progress lines consumable by orchestrator UI.
|
|
|
|
## Notes
|
|
|
|
- Uses Playwright + Crawlee via Camoufox.
|
|
- Low concurrency and longer timeouts for stability.
|
|
- Employer logo (`figure img` alt text) and employer link are optional. Cards that omit them used to throw a Playwright `getAttribute` timeout and fail the whole region page (0 jobs saved even after cards were counted). Missing employer fields are now skipped per card.
|
|
|
|
## Common problems
|
|
|
|
### Region pages time out / run returns 0 jobs
|
|
|
|
- Gradcracker Livewire markup sometimes omits the employer logo on a card. Older builds aborted the whole page on that timeout. Current builds continue and leave employer blank.
|
|
- Anti-bot / Camoufox issues: ensure Camoufox is installed (`npx camoufox-js fetch`) and prefer headed runs when debugging.
|