Slim product to Jobs+Settings and drop expired/rediscovered noise.
Hide unused nav surfaces, probe listing URLs before import, add Danger Zone cleanup for archive matches and dead links, and stop rebuilding jobs on every migrate boot.
This commit is contained in:
parent
529f0a7b02
commit
2c930ba40f
@ -71,6 +71,7 @@ Defaults and constraints:
|
||||
- Every card carries Google's own `data-share-url` permalink for that specific listing, which the extractor uses as `jobUrl` — so a job stays unique and reviewable even if the "click into the card for a detail pane" step below fails for it.
|
||||
- Because resolving a real application link requires clicking each card in a real browser, this extractor is slower per job than API-backed sources — keep `googleJobsMaxJobsPerTerm` modest for frequent runs.
|
||||
- If no external "Apply on <site>" link is found for a card, the extractor falls back to Google's own permalink for that listing so it's still reviewable, but not directly one-click-applyable.
|
||||
- After scraping, JobOps probes each apply URL (preferring the external link over Google's share permalink). Listings that return **404/410** or pages that say the job expired / is no longer available are **dropped before import**. Network failures fail-open (job is kept).
|
||||
- Automated, frequent, or high-volume scraping of Google search results may be against Google's Terms of Service; treat this extractor as best-effort and keep run frequency/volume conservative.
|
||||
- See [Dealing with Google's CAPTCHA / bot detection](#dealing-with-googles-captcha--bot-detection) above for proxy and interactive-solve options.
|
||||
|
||||
|
||||
@ -53,13 +53,23 @@ Use **Filters → Employer keywords → Hide roles you already skipped** to togg
|
||||
|
||||
Uncheck the filter to temporarily review rediscovered rows that match older skips.
|
||||
|
||||
### Clean up rediscovered rows already in Discovered
|
||||
|
||||
If older pipeline runs left Discovered copies of roles you already skipped or applied:
|
||||
|
||||
1. Open **Settings → Danger Zone**.
|
||||
2. Use **Clear Rediscovered Archive Matches**.
|
||||
3. Confirm. JobOps permanently deletes matching **Discovered** rows only (Ready / Applied / Skipped stay).
|
||||
|
||||
Matching uses the same employer+title (and employer+description) keys as skip/import dedup.
|
||||
|
||||
## Defaults and constraints
|
||||
|
||||
- Prior-skip hiding uses employer + title only (not description text).
|
||||
- The 90-day window applies to both the Jobs list filter and pipeline import suppression for skipped/applied rows.
|
||||
- Description matching during import still requires at least **80 characters** of normalized text; short or empty descriptions fall back to employer+title only.
|
||||
- Matching is **per profile** (`ownerProfileId`); different login profiles do not share dedup state.
|
||||
- Dedup does **not** delete existing rows retroactively when you change skip list or country filters — run discovery again or skip manually for old data.
|
||||
- Dedup does **not** delete existing rows retroactively when you change skip list or country filters — use **Clear Rediscovered Archive Matches** in Danger Zone, run discovery again, or skip manually for old data.
|
||||
- Very different titles at the same company (for example `SDET` vs `Product Designer`) are **not** collapsed.
|
||||
|
||||
## Common problems
|
||||
|
||||
@ -186,6 +186,8 @@ Readiness requires:
|
||||
|
||||
|
||||
- Clear jobs by selected statuses
|
||||
- Clear rediscovered archive matches (Discovered jobs that match skipped/applied roles)
|
||||
- Mark expired listings (probe Discovered URLs; mark 404/410 / “job expired” as expired)
|
||||
- Clear jobs below a score threshold
|
||||
- Clear the full database
|
||||
|
||||
|
||||
@ -146,7 +146,21 @@ export async function runGoogleJobs(
|
||||
jobs.push(mapped);
|
||||
}
|
||||
|
||||
return { success: true, jobs };
|
||||
const { filterExpiredJobListings } = await import(
|
||||
"@shared/job-listing-probe.js"
|
||||
);
|
||||
const { kept, dropped } = await filterExpiredJobListings(jobs, {
|
||||
concurrency: 4,
|
||||
timeoutMs: 5_000,
|
||||
});
|
||||
|
||||
if (dropped > 0) {
|
||||
console.info(
|
||||
`[google-jobs] Dropped ${dropped} expired listing(s) after URL probe`,
|
||||
);
|
||||
}
|
||||
|
||||
return { success: true, jobs: kept };
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : "Unknown error";
|
||||
return { success: false, jobs: [], error: message };
|
||||
|
||||
@ -9,7 +9,6 @@
|
||||
<link rel="preconnect" href="https://fonts.googleapis.com">
|
||||
<link rel="preconnect" href="https://fonts.gstatic.com" crossorigin>
|
||||
<link href="https://fonts.googleapis.com/css2?family=Inter:wght@400;500;600;700&display=swap" rel="stylesheet">
|
||||
<script defer src="https://umami.dakheera47.com/script.js" data-website-id="0dc42ed1-87c3-4ac0-9409-5a9b9588fe66"></script>
|
||||
</head>
|
||||
<body>
|
||||
<div id="root"></div>
|
||||
|
||||
@ -91,7 +91,7 @@ describe("App demo banner", () => {
|
||||
});
|
||||
|
||||
render(
|
||||
<MemoryRouter initialEntries={["/overview"]}>
|
||||
<MemoryRouter initialEntries={["/jobs/discovered"]}>
|
||||
<App />
|
||||
</MemoryRouter>,
|
||||
);
|
||||
@ -114,7 +114,7 @@ describe("App demo banner", () => {
|
||||
});
|
||||
|
||||
render(
|
||||
<MemoryRouter initialEntries={["/overview"]}>
|
||||
<MemoryRouter initialEntries={["/jobs/discovered"]}>
|
||||
<App />
|
||||
</MemoryRouter>,
|
||||
);
|
||||
@ -135,7 +135,7 @@ describe("App demo banner", () => {
|
||||
});
|
||||
|
||||
render(
|
||||
<MemoryRouter initialEntries={["/overview"]}>
|
||||
<MemoryRouter initialEntries={["/jobs/discovered"]}>
|
||||
<App />
|
||||
</MemoryRouter>,
|
||||
);
|
||||
|
||||
@ -14,29 +14,31 @@ import { BasicAuthPrompt } from "./components/BasicAuthPrompt";
|
||||
import { OnboardingGate } from "./components/OnboardingGate";
|
||||
import { useDemoInfo } from "./hooks/useDemoInfo";
|
||||
import { GmailOauthCallbackPage } from "./pages/GmailOauthCallbackPage";
|
||||
import { HomePage } from "./pages/HomePage";
|
||||
import { InProgressBoardPage } from "./pages/InProgressBoardPage";
|
||||
import { JobPage } from "./pages/JobPage";
|
||||
import { OrchestratorPage } from "./pages/OrchestratorPage";
|
||||
import { SettingsPage } from "./pages/SettingsPage";
|
||||
import { TracerLinksPage } from "./pages/TracerLinksPage";
|
||||
import { TrackingInboxPage } from "./pages/TrackingInboxPage";
|
||||
import { VisaSponsorsPage } from "./pages/VisaSponsorsPage";
|
||||
|
||||
/** Backwards-compatibility redirects: old URL paths -> new URL paths */
|
||||
const REDIRECTS: Array<{ from: string; to: string }> = [
|
||||
{ from: "/", to: "/jobs/ready" },
|
||||
{ from: "/home", to: "/overview" },
|
||||
{ from: "/", to: "/jobs/discovered" },
|
||||
{ from: "/home", to: "/jobs/discovered" },
|
||||
{ from: "/overview", to: "/jobs/discovered" },
|
||||
{ from: "/ready", to: "/jobs/ready" },
|
||||
{ from: "/ready/:jobId", to: "/jobs/ready/:jobId" },
|
||||
{ from: "/discovered", to: "/jobs/discovered" },
|
||||
{ from: "/discovered/:jobId", to: "/jobs/discovered/:jobId" },
|
||||
{ from: "/applied", to: "/jobs/applied" },
|
||||
{ from: "/applied/:jobId", to: "/jobs/applied/:jobId" },
|
||||
{ from: "/in-progress", to: "/applications/in-progress" },
|
||||
{ from: "/in-progress/:jobId", to: "/applications/in-progress" },
|
||||
{ from: "/jobs/in_progress", to: "/applications/in-progress" },
|
||||
{ from: "/jobs/in_progress/:jobId", to: "/applications/in-progress" },
|
||||
// Slim product: kanban / inbox / tracer / visa browser redirect into Jobs.
|
||||
{ from: "/in-progress", to: "/jobs/discovered" },
|
||||
{ from: "/in-progress/:jobId", to: "/jobs/discovered" },
|
||||
{ from: "/jobs/in_progress", to: "/jobs/discovered" },
|
||||
{ from: "/jobs/in_progress/:jobId", to: "/jobs/discovered" },
|
||||
{ from: "/applications/in-progress", to: "/jobs/discovered" },
|
||||
{ from: "/applications/in-progress/:jobId", to: "/jobs/discovered" },
|
||||
{ from: "/tracking-inbox", to: "/jobs/discovered" },
|
||||
{ from: "/tracer-links", to: "/jobs/discovered" },
|
||||
{ from: "/visa-sponsors", to: "/jobs/discovered" },
|
||||
{ from: "/all", to: "/jobs/all" },
|
||||
{ from: "/all/:jobId", to: "/jobs/all/:jobId" },
|
||||
];
|
||||
@ -134,24 +136,13 @@ export const App: React.FC = () => {
|
||||
/>
|
||||
))}
|
||||
|
||||
{/* Application routes */}
|
||||
<Route path="/overview" element={<HomePage />} />
|
||||
{/* Application routes — slim product: Jobs + Settings */}
|
||||
<Route
|
||||
path="/oauth/gmail/callback"
|
||||
element={<GmailOauthCallbackPage />}
|
||||
/>
|
||||
<Route path="/job/:id" element={<JobPage />} />
|
||||
<Route
|
||||
path="/applications/in-progress"
|
||||
element={<InProgressBoardPage />}
|
||||
/>
|
||||
<Route path="/settings" element={<SettingsPage />} />
|
||||
<Route path="/tracer-links" element={<TracerLinksPage />} />
|
||||
<Route path="/visa-sponsors" element={<VisaSponsorsPage />} />
|
||||
<Route
|
||||
path="/tracking-inbox"
|
||||
element={<TrackingInboxPage />}
|
||||
/>
|
||||
<Route path="/jobs/:tab" element={<OrchestratorPage />} />
|
||||
<Route
|
||||
path="/jobs/:tab/:jobId"
|
||||
|
||||
@ -1618,6 +1618,30 @@ export async function deleteJobsByStatus(status: string): Promise<{
|
||||
});
|
||||
}
|
||||
|
||||
export async function deleteDiscoveredMatchingArchived(): Promise<{
|
||||
message: string;
|
||||
count: number;
|
||||
}> {
|
||||
return fetchApi<{
|
||||
message: string;
|
||||
count: number;
|
||||
}>("/jobs/duplicates/discovered", {
|
||||
method: "DELETE",
|
||||
});
|
||||
}
|
||||
|
||||
export async function expireDeadListings(): Promise<{
|
||||
message: string;
|
||||
count: number;
|
||||
}> {
|
||||
return fetchApi<{
|
||||
message: string;
|
||||
count: number;
|
||||
}>("/jobs/expire-dead-listings", {
|
||||
method: "POST",
|
||||
});
|
||||
}
|
||||
|
||||
export async function deleteJobsBelowScore(threshold: number): Promise<{
|
||||
message: string;
|
||||
count: number;
|
||||
|
||||
@ -51,7 +51,6 @@ import { useRescoreJob } from "../hooks/useRescoreJob";
|
||||
import { FitAssessment, JobHeader, JobNotes, TailoredSummary } from ".";
|
||||
import { CoverLetterDisplay } from "./CoverLetterDisplay";
|
||||
import { TailorMode } from "./discovered-panel/TailorMode";
|
||||
import { GhostwriterDrawer } from "./ghostwriter/GhostwriterDrawer";
|
||||
import { JobDetailsEditDrawer } from "./JobDetailsEditDrawer";
|
||||
import { KbdHint } from "./KbdHint";
|
||||
import { OpenJobListingButton } from "./OpenJobListingButton";
|
||||
@ -380,11 +379,6 @@ export const ReadyPanel: React.FC<ReadyPanelProps> = ({
|
||||
───────────────────────────────────────────────────────────────────── */}
|
||||
<div className="pb-4 border-b border-border/40">
|
||||
<div className="grid gap-2 sm:grid-cols-2">
|
||||
<GhostwriterDrawer
|
||||
job={job}
|
||||
triggerClassName="h-9 w-full justify-center gap-1 px-2 text-xs"
|
||||
/>
|
||||
|
||||
{/* Download PDF - primary artifact action */}
|
||||
<Button
|
||||
asChild
|
||||
|
||||
@ -1,24 +1,20 @@
|
||||
import {
|
||||
Columns3,
|
||||
Home,
|
||||
Inbox,
|
||||
LayoutDashboard,
|
||||
Link2,
|
||||
Settings,
|
||||
Shield,
|
||||
} from "lucide-react";
|
||||
import { LayoutDashboard, Settings } from "lucide-react";
|
||||
|
||||
export type NavLink = {
|
||||
to: string;
|
||||
label: string;
|
||||
icon: typeof Home;
|
||||
icon: typeof LayoutDashboard;
|
||||
activePaths?: string[];
|
||||
};
|
||||
|
||||
/**
|
||||
* Slim product nav: pipeline + scoring only.
|
||||
* Overview / kanban / inbox / tracer / visa browser stay routable for
|
||||
* bookmarks but are intentionally omitted from the chrome.
|
||||
*/
|
||||
export const NAV_LINKS: NavLink[] = [
|
||||
{ to: "/overview", label: "Overview", icon: Home },
|
||||
{
|
||||
to: "/jobs/ready",
|
||||
to: "/jobs/discovered",
|
||||
label: "Jobs",
|
||||
icon: LayoutDashboard,
|
||||
activePaths: [
|
||||
@ -28,20 +24,6 @@ export const NAV_LINKS: NavLink[] = [
|
||||
"/jobs/all",
|
||||
],
|
||||
},
|
||||
{
|
||||
to: "/applications/in-progress",
|
||||
label: "In Progress",
|
||||
icon: Columns3,
|
||||
activePaths: ["/applications/in-progress"],
|
||||
},
|
||||
{ to: "/tracking-inbox", label: "Tracking Inbox", icon: Inbox },
|
||||
{
|
||||
to: "/tracer-links",
|
||||
label: "Tracer Links",
|
||||
icon: Link2,
|
||||
activePaths: ["/tracer-links"],
|
||||
},
|
||||
{ to: "/visa-sponsors", label: "Visa Sponsors", icon: Shield },
|
||||
{ to: "/settings", label: "Settings", icon: Settings },
|
||||
];
|
||||
|
||||
|
||||
@ -55,7 +55,6 @@ import {
|
||||
} from "@/lib/utils";
|
||||
import * as api from "../api";
|
||||
import { ConfirmDelete } from "../components/ConfirmDelete";
|
||||
import { GhostwriterDrawer } from "../components/ghostwriter/GhostwriterDrawer";
|
||||
import { JobDetailsEditDrawer } from "../components/JobDetailsEditDrawer";
|
||||
import { JobHeader } from "../components/JobHeader";
|
||||
import {
|
||||
@ -587,7 +586,6 @@ export const JobPage: React.FC = () => {
|
||||
<CalendarClock className="h-4 w-4" />
|
||||
Application details
|
||||
</CardTitle>
|
||||
<GhostwriterDrawer job={job} />
|
||||
</div>
|
||||
</CardHeader>
|
||||
<CardContent className="space-y-4">
|
||||
|
||||
@ -75,7 +75,7 @@ export const OrchestratorPage: React.FC = () => {
|
||||
if (tab && validTabs.includes(tab as FilterTab)) {
|
||||
return tab as FilterTab;
|
||||
}
|
||||
return "ready";
|
||||
return "discovered";
|
||||
}, [tab]);
|
||||
|
||||
// Helper to change URL while preserving search params
|
||||
@ -96,7 +96,7 @@ export const OrchestratorPage: React.FC = () => {
|
||||
// Effect to sync URL if it was invalid
|
||||
useEffect(() => {
|
||||
if (tab === "in_progress") {
|
||||
navigate("/applications/in-progress", { replace: true });
|
||||
navigate("/jobs/discovered", { replace: true });
|
||||
return;
|
||||
}
|
||||
const validTabs: FilterTab[] = ["ready", "discovered", "applied", "all"];
|
||||
|
||||
@ -2,9 +2,8 @@ import { createAppSettings } from "@shared/testing/factories.js";
|
||||
import { fireEvent, screen, waitFor } from "@testing-library/react";
|
||||
import { MemoryRouter } from "react-router-dom";
|
||||
import { toast } from "sonner";
|
||||
import { beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import { afterAll, beforeEach, describe, expect, it, vi } from "vitest";
|
||||
import * as api from "../api";
|
||||
import { _resetTracerReadinessCache } from "../hooks/useTracerReadiness";
|
||||
import { renderWithQueryClient } from "../test/renderWithQueryClient";
|
||||
import { SettingsPage } from "./SettingsPage";
|
||||
|
||||
@ -19,10 +18,6 @@ vi.mock("../api", () => ({
|
||||
updateSettings: vi.fn(),
|
||||
clearDatabase: vi.fn(),
|
||||
deleteJobsByStatus: vi.fn(),
|
||||
getTracerReadiness: vi.fn(),
|
||||
getBackups: vi.fn().mockResolvedValue({ backups: [], nextScheduled: null }),
|
||||
createManualBackup: vi.fn(),
|
||||
deleteBackup: vi.fn(),
|
||||
}));
|
||||
|
||||
vi.mock("sonner", () => ({
|
||||
@ -65,13 +60,6 @@ const openModelSection = async () => {
|
||||
fireEvent.click(modelTrigger);
|
||||
};
|
||||
|
||||
const openWritingStyleSection = async () => {
|
||||
const chatTrigger = await screen.findByRole("button", {
|
||||
name: /writing style & language/i,
|
||||
});
|
||||
fireEvent.click(chatTrigger);
|
||||
};
|
||||
|
||||
describe("SettingsPage", () => {
|
||||
beforeEach(() => {
|
||||
vi.clearAllMocks();
|
||||
@ -79,16 +67,6 @@ describe("SettingsPage", () => {
|
||||
configurable: true,
|
||||
value: vi.fn(),
|
||||
});
|
||||
_resetTracerReadinessCache();
|
||||
vi.mocked(api.getTracerReadiness).mockResolvedValue({
|
||||
status: "ready",
|
||||
canEnable: true,
|
||||
publicBaseUrl: "https://my-jobops.example.com",
|
||||
healthUrl: "https://my-jobops.example.com/health",
|
||||
checkedAt: Date.now(),
|
||||
lastSuccessAt: Date.now(),
|
||||
reason: null,
|
||||
});
|
||||
});
|
||||
|
||||
afterAll(() => {
|
||||
@ -248,7 +226,7 @@ describe("SettingsPage", () => {
|
||||
);
|
||||
});
|
||||
|
||||
it("hides pipeline tuning sections that moved to run modal", async () => {
|
||||
it("hides pipeline tuning and unused settings sections", async () => {
|
||||
vi.mocked(api.getSettings).mockResolvedValue(baseSettings);
|
||||
renderPage();
|
||||
|
||||
@ -265,57 +243,21 @@ describe("SettingsPage", () => {
|
||||
expect(
|
||||
screen.queryByRole("button", { name: /jobspy scraper/i }),
|
||||
).not.toBeInTheDocument();
|
||||
});
|
||||
|
||||
it("enables save button when display setting is changed", async () => {
|
||||
vi.mocked(api.getSettings).mockResolvedValue(baseSettings);
|
||||
renderPage();
|
||||
const saveButton = screen.getByRole("button", { name: /^save$/i });
|
||||
|
||||
const displayTrigger = await screen.findByRole("button", {
|
||||
name: /display settings/i,
|
||||
});
|
||||
fireEvent.click(displayTrigger);
|
||||
const sponsorCheckbox = screen.getByLabelText(
|
||||
/show visa sponsor information/i,
|
||||
);
|
||||
fireEvent.click(sponsorCheckbox);
|
||||
await waitFor(() => expect(saveButton).toBeEnabled());
|
||||
});
|
||||
|
||||
it("saves the writing language mode through the settings page", async () => {
|
||||
vi.mocked(api.getSettings).mockResolvedValue(baseSettings);
|
||||
vi.mocked(api.updateSettings).mockResolvedValue(
|
||||
createAppSettings({
|
||||
chatStyleLanguageMode: {
|
||||
value: "match-resume",
|
||||
default: "manual",
|
||||
override: "match-resume",
|
||||
},
|
||||
}),
|
||||
);
|
||||
|
||||
renderPage();
|
||||
await openWritingStyleSection();
|
||||
|
||||
fireEvent.click(screen.getByRole("combobox", { name: /output language/i }));
|
||||
fireEvent.click(await screen.findByText("Match current resume language"));
|
||||
|
||||
expect(
|
||||
screen.queryByRole("combobox", { name: /specific language/i }),
|
||||
screen.queryByRole("button", { name: /display settings/i }),
|
||||
).not.toBeInTheDocument();
|
||||
expect(
|
||||
screen.queryByRole("button", { name: /writing style & language/i }),
|
||||
).not.toBeInTheDocument();
|
||||
expect(
|
||||
screen.queryByRole("button", { name: /backup/i }),
|
||||
).not.toBeInTheDocument();
|
||||
expect(
|
||||
screen.queryByRole("button", { name: /tracer/i }),
|
||||
).not.toBeInTheDocument();
|
||||
expect(
|
||||
screen.queryByRole("button", { name: /resume projects/i }),
|
||||
).not.toBeInTheDocument();
|
||||
|
||||
const saveButton = screen.getByRole("button", { name: /^save$/i });
|
||||
await waitFor(() => expect(saveButton).toBeEnabled());
|
||||
fireEvent.click(saveButton);
|
||||
|
||||
await waitFor(() => expect(api.updateSettings).toHaveBeenCalled());
|
||||
expect(api.updateSettings).toHaveBeenCalledWith(
|
||||
expect.objectContaining({
|
||||
chatStyleLanguageMode: "match-resume",
|
||||
chatStyleManualLanguage: null,
|
||||
}),
|
||||
);
|
||||
});
|
||||
|
||||
it("enables save button when basic auth toggle is changed", async () => {
|
||||
|
||||
@ -1,19 +1,13 @@
|
||||
import * as api from "@client/api";
|
||||
import { PageHeader } from "@client/components/layout";
|
||||
import { useUpdateSettingsMutation } from "@client/hooks/queries/useSettingsMutation";
|
||||
import { useTracerReadiness } from "@client/hooks/useTracerReadiness";
|
||||
import { BackupSettingsSection } from "@client/pages/settings/components/BackupSettingsSection";
|
||||
import { ChatSettingsSection } from "@client/pages/settings/components/ChatSettingsSection";
|
||||
import { DangerZoneSection } from "@client/pages/settings/components/DangerZoneSection";
|
||||
import { DisplaySettingsSection } from "@client/pages/settings/components/DisplaySettingsSection";
|
||||
import { EnvironmentSettingsSection } from "@client/pages/settings/components/EnvironmentSettingsSection";
|
||||
import { JobSearchProfileSection } from "@client/pages/settings/components/JobSearchProfileSection";
|
||||
import { JobSourcesSettingsSection } from "@client/pages/settings/components/JobSourcesSettingsSection";
|
||||
import { KeywordSetsSettingsSection } from "@client/pages/settings/components/KeywordSetsSettingsSection";
|
||||
import { ModelSettingsSection } from "@client/pages/settings/components/ModelSettingsSection";
|
||||
import { ResumeSettingsSection } from "@client/pages/settings/components/ResumeSettingsSection";
|
||||
import { ScoringSettingsSection } from "@client/pages/settings/components/ScoringSettingsSection";
|
||||
import { TracerLinksSettingsSection } from "@client/pages/settings/components/TracerLinksSettingsSection";
|
||||
import { WebhooksSection } from "@client/pages/settings/components/WebhooksSection";
|
||||
import {
|
||||
type LlmProviderId,
|
||||
@ -27,22 +21,12 @@ import {
|
||||
type UpdateSettingsInput,
|
||||
updateSettingsSchema,
|
||||
} from "@shared/settings-schema.js";
|
||||
import type {
|
||||
AppSettings,
|
||||
JobStatus,
|
||||
ResumeProjectCatalogItem,
|
||||
ResumeProjectsSettings,
|
||||
} from "@shared/types.js";
|
||||
import { useQuery, useQueryClient } from "@tanstack/react-query";
|
||||
import type { AppSettings, JobStatus } from "@shared/types.js";
|
||||
import { useQuery } from "@tanstack/react-query";
|
||||
import { Settings } from "lucide-react";
|
||||
import type React from "react";
|
||||
import { useCallback, useEffect, useState } from "react";
|
||||
import {
|
||||
FormProvider,
|
||||
type Resolver,
|
||||
useForm,
|
||||
useWatch,
|
||||
} from "react-hook-form";
|
||||
import { useEffect, useState } from "react";
|
||||
import { FormProvider, type Resolver, useForm } from "react-hook-form";
|
||||
import { toast } from "sonner";
|
||||
import { useQueryErrorToast } from "@/client/hooks/useQueryErrorToast";
|
||||
import { queryKeys } from "@/client/lib/queryKeys";
|
||||
@ -236,43 +220,7 @@ const stringArraysEqual = (left: string[], right: string[]): boolean => {
|
||||
const nullIfSame = <T,>(value: T | null | undefined, defaultValue: T) =>
|
||||
value === defaultValue ? null : (value ?? null);
|
||||
|
||||
const normalizeResumeProjectsForCatalog = (
|
||||
catalog: ResumeProjectCatalogItem[],
|
||||
current: ResumeProjectsSettings | null,
|
||||
): ResumeProjectsSettings | null => {
|
||||
const allowed = new Set(catalog.map((project) => project.id));
|
||||
|
||||
const base = current ?? {
|
||||
maxProjects: 0,
|
||||
lockedProjectIds: catalog
|
||||
.filter((project) => project.isVisibleInBase)
|
||||
.map((project) => project.id),
|
||||
aiSelectableProjectIds: [],
|
||||
};
|
||||
|
||||
const lockedProjectIds = base.lockedProjectIds.filter((id) =>
|
||||
allowed.has(id),
|
||||
);
|
||||
const lockedSet = new Set(lockedProjectIds);
|
||||
const aiSelectableProjectIds = (
|
||||
current ? base.aiSelectableProjectIds : catalog.map((project) => project.id)
|
||||
)
|
||||
.filter((id) => allowed.has(id))
|
||||
.filter((id) => !lockedSet.has(id));
|
||||
const maxProjectsRaw = Number.isFinite(base.maxProjects)
|
||||
? base.maxProjects
|
||||
: 0;
|
||||
const maxProjectsInt = Math.max(0, Math.floor(maxProjectsRaw));
|
||||
const maxProjects = Math.min(
|
||||
catalog.length,
|
||||
Math.max(lockedProjectIds.length, maxProjectsInt, 3),
|
||||
);
|
||||
return { maxProjects, lockedProjectIds, aiSelectableProjectIds };
|
||||
};
|
||||
|
||||
const getDerivedSettings = (settings: AppSettings | null) => {
|
||||
const profileProjects = settings?.profileProjects ?? [];
|
||||
|
||||
return {
|
||||
model: {
|
||||
effective: settings?.model?.value ?? "",
|
||||
@ -302,32 +250,6 @@ const getDerivedSettings = (settings: AppSettings | null) => {
|
||||
default: settings?.renderMarkdownInJobDescriptions?.default ?? true,
|
||||
},
|
||||
},
|
||||
chat: {
|
||||
tone: {
|
||||
effective: settings?.chatStyleTone?.value ?? "professional",
|
||||
default: settings?.chatStyleTone?.default ?? "professional",
|
||||
},
|
||||
formality: {
|
||||
effective: settings?.chatStyleFormality?.value ?? "medium",
|
||||
default: settings?.chatStyleFormality?.default ?? "medium",
|
||||
},
|
||||
constraints: {
|
||||
effective: settings?.chatStyleConstraints?.value ?? "",
|
||||
default: settings?.chatStyleConstraints?.default ?? "",
|
||||
},
|
||||
doNotUse: {
|
||||
effective: settings?.chatStyleDoNotUse?.value ?? "",
|
||||
default: settings?.chatStyleDoNotUse?.default ?? "",
|
||||
},
|
||||
languageMode: {
|
||||
effective: settings?.chatStyleLanguageMode?.value ?? "manual",
|
||||
default: settings?.chatStyleLanguageMode?.default ?? "manual",
|
||||
},
|
||||
manualLanguage: {
|
||||
effective: settings?.chatStyleManualLanguage?.value ?? "english",
|
||||
default: settings?.chatStyleManualLanguage?.default ?? "english",
|
||||
},
|
||||
},
|
||||
envSettings: {
|
||||
readable: {
|
||||
ukvisajobsEmail: settings?.ukvisajobsEmail ?? "",
|
||||
@ -343,10 +265,6 @@ const getDerivedSettings = (settings: AppSettings | null) => {
|
||||
basicAuthActive: settings?.basicAuthActive ?? false,
|
||||
},
|
||||
defaultResumeProjects: settings?.resumeProjects?.default ?? null,
|
||||
|
||||
profileProjects,
|
||||
maxProjectsTotal: profileProjects.length,
|
||||
|
||||
backup: {
|
||||
backupEnabled: {
|
||||
effective: settings?.backupEnabled?.value ?? false,
|
||||
@ -441,21 +359,11 @@ const getDerivedSettings = (settings: AppSettings | null) => {
|
||||
};
|
||||
|
||||
export const SettingsPage: React.FC = () => {
|
||||
const queryClient = useQueryClient();
|
||||
const [settings, setSettings] = useState<AppSettings | null>(null);
|
||||
const [isSaving, setIsSaving] = useState(false);
|
||||
const [statusesToClear, setStatusesToClear] = useState<JobStatus[]>([
|
||||
"discovered",
|
||||
]);
|
||||
// Backup state
|
||||
const [isCreatingBackup, setIsCreatingBackup] = useState(false);
|
||||
const [isDeletingBackup, setIsDeletingBackup] = useState(false);
|
||||
const {
|
||||
readiness: tracerReadiness,
|
||||
isLoading: isTracerReadinessLoading,
|
||||
isChecking: isTracerReadinessChecking,
|
||||
refreshReadiness,
|
||||
} = useTracerReadiness();
|
||||
|
||||
const methods = useForm<UpdateSettingsInput>({
|
||||
resolver: zodResolver(
|
||||
@ -466,33 +374,17 @@ export const SettingsPage: React.FC = () => {
|
||||
});
|
||||
|
||||
const {
|
||||
clearErrors,
|
||||
handleSubmit,
|
||||
reset,
|
||||
setError,
|
||||
setValue,
|
||||
control,
|
||||
formState: { isDirty, errors, isValid, dirtyFields },
|
||||
} = methods;
|
||||
const settingsQuery = useQuery({
|
||||
queryKey: queryKeys.settings.current(),
|
||||
queryFn: api.getSettings,
|
||||
});
|
||||
const backupsQuery = useQuery({
|
||||
queryKey: queryKeys.backups.list(),
|
||||
queryFn: api.getBackups,
|
||||
});
|
||||
const updateSettingsMutation = useUpdateSettingsMutation();
|
||||
const isLoading = settingsQuery.isLoading;
|
||||
const backups = backupsQuery.data?.backups ?? [];
|
||||
const nextScheduled = backupsQuery.data?.nextScheduled ?? null;
|
||||
const isLoadingBackups = backupsQuery.isLoading;
|
||||
useQueryErrorToast(backupsQuery.error, "Failed to load backups");
|
||||
|
||||
const resumeProjectsValue = useWatch({
|
||||
control,
|
||||
name: "resumeProjects",
|
||||
});
|
||||
|
||||
useEffect(() => {
|
||||
if (!settingsQuery.data) return;
|
||||
@ -508,77 +400,14 @@ export const SettingsPage: React.FC = () => {
|
||||
pipelineWebhook,
|
||||
jobCompleteWebhook,
|
||||
display,
|
||||
chat,
|
||||
envSettings,
|
||||
defaultResumeProjects,
|
||||
profileProjects,
|
||||
backup,
|
||||
scoring,
|
||||
jobSearchProfile,
|
||||
jobSources,
|
||||
} = derived;
|
||||
|
||||
const handleCreateBackup = async () => {
|
||||
setIsCreatingBackup(true);
|
||||
try {
|
||||
await api.createManualBackup();
|
||||
toast.success("Backup created successfully");
|
||||
await queryClient.invalidateQueries({ queryKey: queryKeys.backups.all });
|
||||
} catch (error) {
|
||||
const message =
|
||||
error instanceof Error ? error.message : "Failed to create backup";
|
||||
toast.error(message);
|
||||
} finally {
|
||||
setIsCreatingBackup(false);
|
||||
}
|
||||
};
|
||||
|
||||
const handleDeleteBackup = async (filename: string) => {
|
||||
const confirmed = window.confirm(
|
||||
`Delete backup "${filename}"? This action cannot be undone.`,
|
||||
);
|
||||
if (!confirmed) {
|
||||
return;
|
||||
}
|
||||
setIsDeletingBackup(true);
|
||||
try {
|
||||
await api.deleteBackup(filename);
|
||||
toast.success("Backup deleted successfully");
|
||||
await queryClient.invalidateQueries({ queryKey: queryKeys.backups.all });
|
||||
} catch (error) {
|
||||
const message =
|
||||
error instanceof Error ? error.message : "Failed to delete backup";
|
||||
toast.error(message);
|
||||
} finally {
|
||||
setIsDeletingBackup(false);
|
||||
}
|
||||
};
|
||||
|
||||
const handleVerifyTracerReadiness = useCallback(async () => {
|
||||
try {
|
||||
const readiness = await refreshReadiness(true);
|
||||
if (!readiness) {
|
||||
toast.error("Tracer links are unavailable. Verify your public URL.");
|
||||
} else if (readiness.canEnable) {
|
||||
toast.success("Tracer links are ready");
|
||||
} else {
|
||||
toast.error(
|
||||
readiness.reason ??
|
||||
"Tracer links are unavailable. Verify your public URL.",
|
||||
);
|
||||
}
|
||||
} catch (error) {
|
||||
const message =
|
||||
error instanceof Error
|
||||
? error.message
|
||||
: "Failed to verify tracer-link readiness";
|
||||
toast.error(message);
|
||||
}
|
||||
}, [refreshReadiness]);
|
||||
|
||||
const effectiveMaxProjectsTotal = profileProjects.length;
|
||||
const lockedCount = resumeProjectsValue?.lockedProjectIds.length ?? 0;
|
||||
|
||||
const canSave = isDirty && isValid;
|
||||
|
||||
const onSave = async (data: UpdateSettingsInput) => {
|
||||
@ -959,6 +788,57 @@ export const SettingsPage: React.FC = () => {
|
||||
}
|
||||
};
|
||||
|
||||
const handleClearDiscoveredMatchingArchived = async () => {
|
||||
try {
|
||||
setIsSaving(true);
|
||||
const result = await api.deleteDiscoveredMatchingArchived();
|
||||
|
||||
if (result.count > 0) {
|
||||
toast.success("Rediscovered matches cleared", {
|
||||
description: `Deleted ${result.count} discovered job${result.count === 1 ? "" : "s"} matching skipped or applied roles.`,
|
||||
});
|
||||
} else {
|
||||
toast.info("No matches found", {
|
||||
description:
|
||||
"No discovered jobs matched skipped or applied archive roles.",
|
||||
});
|
||||
}
|
||||
} catch (error) {
|
||||
const message =
|
||||
error instanceof Error
|
||||
? error.message
|
||||
: "Failed to clear rediscovered archive matches";
|
||||
toast.error(message);
|
||||
} finally {
|
||||
setIsSaving(false);
|
||||
}
|
||||
};
|
||||
|
||||
const handleExpireDeadListings = async () => {
|
||||
try {
|
||||
setIsSaving(true);
|
||||
const result = await api.expireDeadListings();
|
||||
|
||||
if (result.count > 0) {
|
||||
toast.success("Expired listings marked", {
|
||||
description: `Marked ${result.count} discovered job${result.count === 1 ? "" : "s"} as expired.`,
|
||||
});
|
||||
} else {
|
||||
toast.info("No expired listings found", {
|
||||
description: "All probed discovered job links still look live.",
|
||||
});
|
||||
}
|
||||
} catch (error) {
|
||||
const message =
|
||||
error instanceof Error
|
||||
? error.message
|
||||
: "Failed to mark expired listings";
|
||||
toast.error(message);
|
||||
} finally {
|
||||
setIsSaving(false);
|
||||
}
|
||||
};
|
||||
|
||||
const toggleStatusToClear = (status: JobStatus) => {
|
||||
setStatusesToClear((prev) =>
|
||||
prev.includes(status)
|
||||
@ -999,36 +879,8 @@ export const SettingsPage: React.FC = () => {
|
||||
isLoading={isLoading}
|
||||
isSaving={isSaving}
|
||||
/>
|
||||
<WebhooksSection
|
||||
pipelineWebhook={pipelineWebhook}
|
||||
jobCompleteWebhook={jobCompleteWebhook}
|
||||
webhookSecretHint={envSettings.private.webhookSecretHint}
|
||||
isLoading={isLoading}
|
||||
isSaving={isSaving}
|
||||
/>
|
||||
<ResumeSettingsSection
|
||||
profileProjects={profileProjects}
|
||||
lockedCount={lockedCount}
|
||||
maxProjectsTotal={effectiveMaxProjectsTotal}
|
||||
localResumeFileConfigured={Boolean(
|
||||
settings?.localResumeFileConfigured,
|
||||
)}
|
||||
isLoading={isLoading}
|
||||
isSaving={isSaving}
|
||||
/>
|
||||
<TracerLinksSettingsSection
|
||||
readiness={tracerReadiness}
|
||||
isLoading={isLoading || isTracerReadinessLoading}
|
||||
isChecking={isTracerReadinessChecking}
|
||||
onVerifyNow={handleVerifyTracerReadiness}
|
||||
/>
|
||||
<DisplaySettingsSection
|
||||
values={display}
|
||||
isLoading={isLoading}
|
||||
isSaving={isSaving}
|
||||
/>
|
||||
<ChatSettingsSection
|
||||
values={chat}
|
||||
<ScoringSettingsSection
|
||||
values={scoring}
|
||||
isLoading={isLoading}
|
||||
isSaving={isSaving}
|
||||
/>
|
||||
@ -1047,8 +899,10 @@ export const SettingsPage: React.FC = () => {
|
||||
isLoading={isLoading}
|
||||
isSaving={isSaving}
|
||||
/>
|
||||
<ScoringSettingsSection
|
||||
values={scoring}
|
||||
<WebhooksSection
|
||||
pipelineWebhook={pipelineWebhook}
|
||||
jobCompleteWebhook={jobCompleteWebhook}
|
||||
webhookSecretHint={envSettings.private.webhookSecretHint}
|
||||
isLoading={isLoading}
|
||||
isSaving={isSaving}
|
||||
/>
|
||||
@ -1057,23 +911,16 @@ export const SettingsPage: React.FC = () => {
|
||||
isLoading={isLoading}
|
||||
isSaving={isSaving}
|
||||
/>
|
||||
<BackupSettingsSection
|
||||
values={backup}
|
||||
backups={backups}
|
||||
nextScheduled={nextScheduled}
|
||||
isLoading={isLoading || isLoadingBackups}
|
||||
isSaving={isSaving}
|
||||
onCreateBackup={handleCreateBackup}
|
||||
onDeleteBackup={handleDeleteBackup}
|
||||
isCreatingBackup={isCreatingBackup}
|
||||
isDeletingBackup={isDeletingBackup}
|
||||
/>
|
||||
<DangerZoneSection
|
||||
statusesToClear={statusesToClear}
|
||||
toggleStatusToClear={toggleStatusToClear}
|
||||
handleClearByStatuses={handleClearByStatuses}
|
||||
handleClearDatabase={handleClearDatabase}
|
||||
handleClearByScore={handleClearByScore}
|
||||
handleClearDiscoveredMatchingArchived={
|
||||
handleClearDiscoveredMatchingArchived
|
||||
}
|
||||
handleExpireDeadListings={handleExpireDeadListings}
|
||||
isLoading={isLoading}
|
||||
isSaving={isSaving}
|
||||
/>
|
||||
|
||||
@ -14,11 +14,15 @@ export const sponsorshipSignalFilterOptions = [
|
||||
...SPONSORSHIP_SIGNAL_IDS,
|
||||
] as const;
|
||||
|
||||
/** Defaults aligned with the live cron set on jobs.levkin.ca. */
|
||||
export const DEFAULT_PIPELINE_SOURCES: JobSource[] = [
|
||||
"gradcracker",
|
||||
"indeed",
|
||||
"linkedin",
|
||||
"ukvisajobs",
|
||||
"indeed",
|
||||
"glassdoor",
|
||||
"qajobsboard",
|
||||
"arcdev",
|
||||
"eluta",
|
||||
"bctenet",
|
||||
];
|
||||
export const PIPELINE_SOURCES_STORAGE_KEY = "jobops.pipeline.sources";
|
||||
|
||||
|
||||
@ -34,6 +34,8 @@ type DangerZoneSectionProps = {
|
||||
handleClearByStatuses: () => void;
|
||||
handleClearDatabase: () => void;
|
||||
handleClearByScore?: (threshold: number) => void;
|
||||
handleClearDiscoveredMatchingArchived?: () => void;
|
||||
handleExpireDeadListings?: () => void;
|
||||
isLoading: boolean;
|
||||
isSaving: boolean;
|
||||
};
|
||||
@ -44,6 +46,8 @@ export const DangerZoneSection: React.FC<DangerZoneSectionProps> = ({
|
||||
handleClearByStatuses,
|
||||
handleClearDatabase,
|
||||
handleClearByScore,
|
||||
handleClearDiscoveredMatchingArchived,
|
||||
handleExpireDeadListings,
|
||||
isLoading,
|
||||
isSaving,
|
||||
}) => {
|
||||
@ -154,6 +158,106 @@ export const DangerZoneSection: React.FC<DangerZoneSectionProps> = ({
|
||||
|
||||
<Separator />
|
||||
|
||||
{handleClearDiscoveredMatchingArchived && (
|
||||
<div className="flex flex-col gap-4 sm:flex-row sm:items-center sm:justify-between p-3 rounded-md">
|
||||
<div className="space-y-0.5">
|
||||
<div className="text-sm font-semibold text-destructive">
|
||||
Clear Rediscovered Archive Matches
|
||||
</div>
|
||||
<div className="text-xs text-muted-foreground">
|
||||
Delete Discovered jobs that match roles you already skipped or
|
||||
applied (same employer + title, including cross-source
|
||||
reposts).
|
||||
</div>
|
||||
</div>
|
||||
<AlertDialog>
|
||||
<AlertDialogTrigger asChild>
|
||||
<Button
|
||||
variant="destructive"
|
||||
size="sm"
|
||||
disabled={isLoading || isSaving}
|
||||
>
|
||||
<Trash2 className="mr-2 h-4 w-4" />
|
||||
Clear Matches
|
||||
</Button>
|
||||
</AlertDialogTrigger>
|
||||
<AlertDialogContent>
|
||||
<AlertDialogHeader>
|
||||
<AlertDialogTitle>
|
||||
Clear rediscovered archive matches?
|
||||
</AlertDialogTitle>
|
||||
<AlertDialogDescription>
|
||||
This permanently deletes Discovered jobs that match
|
||||
skipped or applied roles. Ready, applied, and skipped jobs
|
||||
are not deleted. This action cannot be undone.
|
||||
</AlertDialogDescription>
|
||||
</AlertDialogHeader>
|
||||
<AlertDialogFooter>
|
||||
<AlertDialogCancel>Cancel</AlertDialogCancel>
|
||||
<AlertDialogAction
|
||||
onClick={handleClearDiscoveredMatchingArchived}
|
||||
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
|
||||
>
|
||||
Clear matches
|
||||
</AlertDialogAction>
|
||||
</AlertDialogFooter>
|
||||
</AlertDialogContent>
|
||||
</AlertDialog>
|
||||
</div>
|
||||
)}
|
||||
|
||||
{handleExpireDeadListings && (
|
||||
<>
|
||||
<Separator />
|
||||
<div className="flex flex-col gap-4 sm:flex-row sm:items-center sm:justify-between p-3 rounded-md">
|
||||
<div className="space-y-0.5">
|
||||
<div className="text-sm font-semibold text-destructive">
|
||||
Mark Expired Listings
|
||||
</div>
|
||||
<div className="text-xs text-muted-foreground">
|
||||
Open each Discovered job URL. If the page is gone (404/410
|
||||
or “job expired”), mark it expired so it leaves the queue.
|
||||
</div>
|
||||
</div>
|
||||
<AlertDialog>
|
||||
<AlertDialogTrigger asChild>
|
||||
<Button
|
||||
variant="destructive"
|
||||
size="sm"
|
||||
disabled={isLoading || isSaving}
|
||||
>
|
||||
<Trash2 className="mr-2 h-4 w-4" />
|
||||
Mark Expired
|
||||
</Button>
|
||||
</AlertDialogTrigger>
|
||||
<AlertDialogContent>
|
||||
<AlertDialogHeader>
|
||||
<AlertDialogTitle>
|
||||
Mark expired listings?
|
||||
</AlertDialogTitle>
|
||||
<AlertDialogDescription>
|
||||
This probes Discovered job links and marks dead ones as
|
||||
expired. Network failures are ignored (jobs are kept).
|
||||
This can take a few minutes for large queues.
|
||||
</AlertDialogDescription>
|
||||
</AlertDialogHeader>
|
||||
<AlertDialogFooter>
|
||||
<AlertDialogCancel>Cancel</AlertDialogCancel>
|
||||
<AlertDialogAction
|
||||
onClick={handleExpireDeadListings}
|
||||
className="bg-destructive text-destructive-foreground hover:bg-destructive/90"
|
||||
>
|
||||
Mark expired
|
||||
</AlertDialogAction>
|
||||
</AlertDialogFooter>
|
||||
</AlertDialogContent>
|
||||
</AlertDialog>
|
||||
</div>
|
||||
</>
|
||||
)}
|
||||
|
||||
<Separator />
|
||||
|
||||
{/* Clear Jobs Below Score */}
|
||||
{handleClearByScore && (
|
||||
<div className="p-3 rounded-md space-y-4">
|
||||
|
||||
@ -860,6 +860,122 @@ describe.sequential("Jobs API routes", () => {
|
||||
expect(remainingJobIds).toContain(appliedLowScoreJob.id); // Applied job preserved
|
||||
});
|
||||
|
||||
it("deletes discovered jobs that match skipped or applied archive roles", async () => {
|
||||
const { createJob, updateJob } = await import("@server/repositories/jobs");
|
||||
const { buildJobContentFingerprint } = await import(
|
||||
"@shared/job-fingerprint"
|
||||
);
|
||||
const { db, schema } = await import("@server/db/index");
|
||||
const { randomUUID } = await import("node:crypto");
|
||||
|
||||
const skipped = await createJob({
|
||||
source: "manual",
|
||||
title: "SDET (Remote)",
|
||||
employer: "Acme Inc.",
|
||||
jobUrl: "https://example.com/job/archive-skipped",
|
||||
jobDescription: "Test description for skipped archive role",
|
||||
});
|
||||
await updateJob(skipped.id, { status: "skipped" });
|
||||
|
||||
const applied = await createJob({
|
||||
source: "manual",
|
||||
title: "Platform Engineer",
|
||||
employer: "Beta Corp",
|
||||
jobUrl: "https://example.com/job/archive-applied",
|
||||
jobDescription: "Test description for applied archive role",
|
||||
});
|
||||
await updateJob(applied.id, { status: "applied" });
|
||||
|
||||
const now = new Date().toISOString();
|
||||
const rediscoveredSkippedId = randomUUID();
|
||||
const rediscoveredAppliedId = randomUUID();
|
||||
const readyMatchId = randomUUID();
|
||||
|
||||
// Insert rediscovered rows directly so createJob dedup does not collapse them.
|
||||
await db.insert(schema.jobs).values([
|
||||
{
|
||||
id: rediscoveredSkippedId,
|
||||
ownerProfileId: skipped.ownerProfileId,
|
||||
source: "linkedin",
|
||||
title: "SDET",
|
||||
employer: "Acme",
|
||||
jobUrl: "https://example.com/job/rediscovered-skipped",
|
||||
jobDescription: "Different board copy of the skipped role",
|
||||
contentFingerprint: buildJobContentFingerprint({
|
||||
employer: "Acme",
|
||||
title: "SDET",
|
||||
}),
|
||||
status: "discovered",
|
||||
discoveredAt: now,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
},
|
||||
{
|
||||
id: rediscoveredAppliedId,
|
||||
ownerProfileId: applied.ownerProfileId,
|
||||
source: "indeed",
|
||||
title: "Platform Engineer - Toronto, ON",
|
||||
employer: "Beta Corporation",
|
||||
jobUrl: "https://example.com/job/rediscovered-applied",
|
||||
jobDescription: "Different board copy of the applied role",
|
||||
contentFingerprint: buildJobContentFingerprint({
|
||||
employer: "Beta Corporation",
|
||||
title: "Platform Engineer - Toronto, ON",
|
||||
}),
|
||||
status: "discovered",
|
||||
discoveredAt: now,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
},
|
||||
{
|
||||
id: readyMatchId,
|
||||
ownerProfileId: skipped.ownerProfileId,
|
||||
source: "manual",
|
||||
title: "SDET",
|
||||
employer: "Acme",
|
||||
jobUrl: "https://example.com/job/ready-match",
|
||||
jobDescription: "Ready sibling should not be deleted",
|
||||
contentFingerprint: buildJobContentFingerprint({
|
||||
employer: "Acme",
|
||||
title: "SDET",
|
||||
}),
|
||||
status: "ready",
|
||||
discoveredAt: now,
|
||||
createdAt: now,
|
||||
updatedAt: now,
|
||||
},
|
||||
]);
|
||||
|
||||
const uniqueDiscovered = await createJob({
|
||||
source: "manual",
|
||||
title: "Unrelated Designer",
|
||||
employer: "Gamma",
|
||||
jobUrl: "https://example.com/job/unique-discovered",
|
||||
jobDescription: "Should remain after cleanup",
|
||||
});
|
||||
|
||||
const deleteRes = await fetch(`${baseUrl}/api/jobs/duplicates/discovered`, {
|
||||
method: "DELETE",
|
||||
});
|
||||
const deleteBody = await deleteRes.json();
|
||||
|
||||
expect(deleteRes.status).toBe(200);
|
||||
expect(deleteBody.ok).toBe(true);
|
||||
expect(deleteBody.data.count).toBe(2);
|
||||
expect(typeof deleteBody.meta.requestId).toBe("string");
|
||||
|
||||
const listRes = await fetch(`${baseUrl}/api/jobs`);
|
||||
const listBody = await listRes.json();
|
||||
const remainingIds = listBody.data.jobs.map((j: { id: string }) => j.id);
|
||||
|
||||
expect(remainingIds).not.toContain(rediscoveredSkippedId);
|
||||
expect(remainingIds).not.toContain(rediscoveredAppliedId);
|
||||
expect(remainingIds).toContain(skipped.id);
|
||||
expect(remainingIds).toContain(applied.id);
|
||||
expect(remainingIds).toContain(uniqueDiscovered.id);
|
||||
expect(remainingIds).toContain(readyMatchId);
|
||||
});
|
||||
|
||||
it("rejects invalid score thresholds", async () => {
|
||||
// Test invalid threshold (above 100)
|
||||
const invalidRes = await fetch(`${baseUrl}/api/jobs/score/150`, {
|
||||
|
||||
@ -1447,6 +1447,109 @@ jobsRouter.post("/:id/apply", async (req: Request, res: Response) => {
|
||||
}
|
||||
});
|
||||
|
||||
/**
|
||||
* DELETE /api/jobs/duplicates/discovered - Remove discovered jobs that match
|
||||
* skipped/applied archive rows (same employer+title / description keys).
|
||||
*/
|
||||
jobsRouter.delete(
|
||||
"/duplicates/discovered",
|
||||
async (_req: Request, res: Response) => {
|
||||
const requestId = String(res.getHeader("x-request-id") || "unknown");
|
||||
try {
|
||||
if (isDemoMode()) {
|
||||
return sendDemoBlocked(
|
||||
res,
|
||||
"Cleaning discovered duplicates is disabled to keep the demo stable.",
|
||||
{ route: "DELETE /api/jobs/duplicates/discovered" },
|
||||
);
|
||||
}
|
||||
|
||||
const count = await jobsRepo.deleteDiscoveredJobsMatchingArchived();
|
||||
|
||||
logger.info("Discovered archive duplicates cleaned", {
|
||||
requestId,
|
||||
route: "DELETE /api/jobs/duplicates/discovered",
|
||||
count,
|
||||
});
|
||||
|
||||
return res.json({
|
||||
ok: true,
|
||||
data: {
|
||||
message: `Deleted ${count} discovered job${count === 1 ? "" : "s"} matching skipped or applied roles`,
|
||||
count,
|
||||
},
|
||||
meta: { requestId },
|
||||
});
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : "Unknown error";
|
||||
logger.error("Failed to clean discovered archive duplicates", {
|
||||
requestId,
|
||||
route: "DELETE /api/jobs/duplicates/discovered",
|
||||
error: message,
|
||||
});
|
||||
return res.status(500).json({
|
||||
ok: false,
|
||||
error: {
|
||||
code: "INTERNAL_ERROR",
|
||||
message,
|
||||
},
|
||||
meta: { requestId },
|
||||
});
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
/**
|
||||
* POST /api/jobs/expire-dead-listings - Probe discovered URLs and mark expired.
|
||||
*/
|
||||
jobsRouter.post(
|
||||
"/expire-dead-listings",
|
||||
async (_req: Request, res: Response) => {
|
||||
const requestId = String(res.getHeader("x-request-id") || "unknown");
|
||||
try {
|
||||
if (isDemoMode()) {
|
||||
return sendDemoBlocked(
|
||||
res,
|
||||
"Expiring dead listings is disabled to keep the demo stable.",
|
||||
{ route: "POST /api/jobs/expire-dead-listings" },
|
||||
);
|
||||
}
|
||||
|
||||
const count = await jobsRepo.expireDiscoveredJobsWithDeadListings();
|
||||
|
||||
logger.info("Expired discovered jobs with dead listing URLs", {
|
||||
requestId,
|
||||
route: "POST /api/jobs/expire-dead-listings",
|
||||
count,
|
||||
});
|
||||
|
||||
return res.json({
|
||||
ok: true,
|
||||
data: {
|
||||
message: `Marked ${count} discovered job${count === 1 ? "" : "s"} as expired`,
|
||||
count,
|
||||
},
|
||||
meta: { requestId },
|
||||
});
|
||||
} catch (error) {
|
||||
const message = error instanceof Error ? error.message : "Unknown error";
|
||||
logger.error("Failed to expire dead listings", {
|
||||
requestId,
|
||||
route: "POST /api/jobs/expire-dead-listings",
|
||||
error: message,
|
||||
});
|
||||
return res.status(500).json({
|
||||
ok: false,
|
||||
error: {
|
||||
code: "INTERNAL_ERROR",
|
||||
message,
|
||||
},
|
||||
meta: { requestId },
|
||||
});
|
||||
}
|
||||
},
|
||||
);
|
||||
|
||||
/**
|
||||
* DELETE /api/jobs/status/:status - Clear jobs with a specific status
|
||||
*/
|
||||
|
||||
@ -1146,7 +1146,50 @@ const migrations = [
|
||||
|
||||
console.log("🔧 Running database migrations...");
|
||||
|
||||
sqlite.exec(`
|
||||
CREATE TABLE IF NOT EXISTS schema_migrations (
|
||||
id TEXT PRIMARY KEY,
|
||||
applied_at TEXT NOT NULL DEFAULT (datetime('now'))
|
||||
)
|
||||
`);
|
||||
|
||||
const jobsReownerApplied = Boolean(
|
||||
(
|
||||
sqlite
|
||||
.prepare(`SELECT 1 AS ok FROM schema_migrations WHERE id = ?`)
|
||||
.get("jobs_reowner_v1") as { ok?: number } | undefined
|
||||
)?.ok,
|
||||
);
|
||||
|
||||
const jobsTableSql = (
|
||||
sqlite
|
||||
.prepare(
|
||||
`SELECT sql AS sql FROM sqlite_master WHERE type = 'table' AND name = 'jobs'`,
|
||||
)
|
||||
.get() as { sql?: string } | undefined
|
||||
)?.sql;
|
||||
const jobsAlreadyReowned =
|
||||
jobsReownerApplied ||
|
||||
Boolean(jobsTableSql?.includes("UNIQUE(owner_profile_id, job_url)"));
|
||||
|
||||
const REOWNER_MIGRATION_MARKERS = [
|
||||
"DROP TABLE IF EXISTS jobs_reowner_v1",
|
||||
"CREATE TABLE jobs_reowner_v1 (",
|
||||
"INSERT OR REPLACE INTO jobs_reowner_v1 (",
|
||||
"ALTER TABLE jobs_reowner_v1 RENAME TO jobs",
|
||||
] as const;
|
||||
|
||||
for (const migration of migrations) {
|
||||
// One-time table rebuild: do not DROP/recreate jobs on every boot.
|
||||
if (
|
||||
jobsAlreadyReowned &&
|
||||
(REOWNER_MIGRATION_MARKERS.some((marker) => migration.includes(marker)) ||
|
||||
migration.trim() === "DROP TABLE jobs")
|
||||
) {
|
||||
console.log("↩️ Migration skipped (jobs already reowned)");
|
||||
continue;
|
||||
}
|
||||
|
||||
try {
|
||||
sqlite.exec(migration);
|
||||
console.log("✅ Migration applied");
|
||||
@ -1197,5 +1240,22 @@ for (const migration of migrations) {
|
||||
}
|
||||
}
|
||||
|
||||
if (!jobsReownerApplied) {
|
||||
const sqlAfter = (
|
||||
sqlite
|
||||
.prepare(
|
||||
`SELECT sql AS sql FROM sqlite_master WHERE type = 'table' AND name = 'jobs'`,
|
||||
)
|
||||
.get() as { sql?: string } | undefined
|
||||
)?.sql;
|
||||
if (sqlAfter?.includes("UNIQUE(owner_profile_id, job_url)")) {
|
||||
sqlite
|
||||
.prepare(
|
||||
`INSERT OR IGNORE INTO schema_migrations (id, applied_at) VALUES (?, datetime('now'))`,
|
||||
)
|
||||
.run("jobs_reowner_v1");
|
||||
}
|
||||
}
|
||||
|
||||
sqlite.close();
|
||||
console.log("🎉 Database migrations complete!");
|
||||
|
||||
@ -42,12 +42,21 @@ const DEFAULT_CONFIG: PipelineConfig = {
|
||||
topN: 10,
|
||||
minSuitabilityScore: 50,
|
||||
// Glassdoor runs inside JobSpy with Indeed/LinkedIn; upstream often errors without failing the run.
|
||||
sources: ["gradcracker", "indeed", "linkedin", "glassdoor", "ukvisajobs"],
|
||||
sources: [
|
||||
"linkedin",
|
||||
"indeed",
|
||||
"glassdoor",
|
||||
"qajobsboard",
|
||||
"arcdev",
|
||||
"eluta",
|
||||
"bctenet",
|
||||
],
|
||||
outputDir: join(getDataDir(), "pdfs"),
|
||||
enableCrawling: true,
|
||||
enableScoring: true,
|
||||
enableImporting: true,
|
||||
enableAutoTailoring: true,
|
||||
// Slim product: discover + score only (no PDF / RxResume tailoring).
|
||||
enableAutoTailoring: false,
|
||||
};
|
||||
|
||||
// Track if pipeline is currently running
|
||||
|
||||
@ -12,6 +12,7 @@ import {
|
||||
jobMatchesBlockedCountries,
|
||||
resolveBlockedCountriesFromStoredString,
|
||||
} from "@shared/blocked-countries.js";
|
||||
import { filterExpiredJobListings } from "@shared/job-listing-probe.js";
|
||||
import {
|
||||
textMatchesAnyKeyword,
|
||||
textMatchesKeyword,
|
||||
@ -742,9 +743,28 @@ export async function discoverJobsStep(args: {
|
||||
logger.warn("Some discovery sources failed", { sourceErrors });
|
||||
}
|
||||
|
||||
progressHelpers.crawlingComplete(profileFiltered.jobs.length);
|
||||
if (args.shouldCancel?.()) {
|
||||
return { discoveredJobs: profileFiltered.jobs, sourceErrors };
|
||||
}
|
||||
|
||||
const stamped = profileFiltered.jobs.map((job) => ({
|
||||
const { kept: liveJobs, dropped: expiredDropped } =
|
||||
await filterExpiredJobListings(profileFiltered.jobs, {
|
||||
concurrency: 4,
|
||||
timeoutMs: 5_000,
|
||||
shouldStop: args.shouldCancel,
|
||||
});
|
||||
|
||||
if (expiredDropped > 0) {
|
||||
logger.info("Dropped discovered jobs with expired listing URLs", {
|
||||
step: "discover-jobs",
|
||||
droppedCount: expiredDropped,
|
||||
keptCount: liveJobs.length,
|
||||
});
|
||||
}
|
||||
|
||||
progressHelpers.crawlingComplete(liveJobs.length);
|
||||
|
||||
const stamped = liveJobs.map((job) => ({
|
||||
...job,
|
||||
ownerProfileId,
|
||||
}));
|
||||
|
||||
@ -67,81 +67,101 @@ export async function scoreJobsStep(args: {
|
||||
task: async (job) => {
|
||||
if (args.shouldCancel?.()) return;
|
||||
|
||||
const hasCachedScore =
|
||||
typeof job.suitabilityScore === "number" &&
|
||||
!Number.isNaN(job.suitabilityScore);
|
||||
try {
|
||||
const hasCachedScore =
|
||||
typeof job.suitabilityScore === "number" &&
|
||||
!Number.isNaN(job.suitabilityScore);
|
||||
|
||||
if (hasCachedScore) {
|
||||
if (job.sponsorshipSignals == null) {
|
||||
await jobsRepo.updateJob(job.id, buildSponsorshipSignalsUpdate(job));
|
||||
if (hasCachedScore) {
|
||||
if (job.sponsorshipSignals == null) {
|
||||
await jobsRepo.updateJob(
|
||||
job.id,
|
||||
buildSponsorshipSignalsUpdate(job),
|
||||
);
|
||||
}
|
||||
completed += 1;
|
||||
progressHelpers.scoringJob(
|
||||
completed,
|
||||
unprocessedJobs.length,
|
||||
`${job.title} (cached)`,
|
||||
);
|
||||
scoredJobs.push({
|
||||
...job,
|
||||
suitabilityScore: job.suitabilityScore as number,
|
||||
suitabilityReason: job.suitabilityReason ?? "",
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
const { score, reason, analysis } = await scoreJobSuitability(
|
||||
job,
|
||||
args.profile,
|
||||
);
|
||||
if (args.shouldCancel?.()) return;
|
||||
|
||||
let sponsorMatchScore = 0;
|
||||
let sponsorMatchNames: string | undefined;
|
||||
|
||||
if (job.employer) {
|
||||
const sponsorResults = await visaSponsors.searchSponsors(
|
||||
job.employer,
|
||||
{
|
||||
limit: 10,
|
||||
minScore: 50,
|
||||
},
|
||||
);
|
||||
|
||||
const summary =
|
||||
visaSponsors.calculateSponsorMatchSummary(sponsorResults);
|
||||
sponsorMatchScore = summary.sponsorMatchScore;
|
||||
sponsorMatchNames = summary.sponsorMatchNames ?? undefined;
|
||||
}
|
||||
|
||||
const shouldAutoSkip =
|
||||
job.status !== "applied" &&
|
||||
autoSkipThreshold !== null &&
|
||||
!Number.isNaN(autoSkipThreshold) &&
|
||||
score < autoSkipThreshold;
|
||||
|
||||
await jobsRepo.updateJob(job.id, {
|
||||
suitabilityScore: score,
|
||||
suitabilityReason: reason,
|
||||
suitabilityAnalysis: analysis ? JSON.stringify(analysis) : undefined,
|
||||
sponsorMatchScore,
|
||||
sponsorMatchNames,
|
||||
...buildSponsorshipSignalsUpdate(job),
|
||||
...(shouldAutoSkip ? { status: "skipped" } : {}),
|
||||
});
|
||||
|
||||
if (shouldAutoSkip) {
|
||||
logger.info("Auto-skipped job due to low score", {
|
||||
jobId: job.id,
|
||||
title: job.title,
|
||||
score,
|
||||
threshold: autoSkipThreshold,
|
||||
});
|
||||
}
|
||||
|
||||
completed += 1;
|
||||
progressHelpers.scoringJob(
|
||||
completed,
|
||||
unprocessedJobs.length,
|
||||
`${job.title} (cached)`,
|
||||
job.title,
|
||||
);
|
||||
scoredJobs.push({
|
||||
...job,
|
||||
suitabilityScore: job.suitabilityScore as number,
|
||||
suitabilityReason: job.suitabilityReason ?? "",
|
||||
suitabilityScore: score,
|
||||
suitabilityReason: reason,
|
||||
});
|
||||
return;
|
||||
}
|
||||
|
||||
const { score, reason, analysis } = await scoreJobSuitability(
|
||||
job,
|
||||
args.profile,
|
||||
);
|
||||
if (args.shouldCancel?.()) return;
|
||||
|
||||
let sponsorMatchScore = 0;
|
||||
let sponsorMatchNames: string | undefined;
|
||||
|
||||
if (job.employer) {
|
||||
const sponsorResults = await visaSponsors.searchSponsors(job.employer, {
|
||||
limit: 10,
|
||||
minScore: 50,
|
||||
});
|
||||
|
||||
const summary =
|
||||
visaSponsors.calculateSponsorMatchSummary(sponsorResults);
|
||||
sponsorMatchScore = summary.sponsorMatchScore;
|
||||
sponsorMatchNames = summary.sponsorMatchNames ?? undefined;
|
||||
}
|
||||
|
||||
const shouldAutoSkip =
|
||||
job.status !== "applied" &&
|
||||
autoSkipThreshold !== null &&
|
||||
!Number.isNaN(autoSkipThreshold) &&
|
||||
score < autoSkipThreshold;
|
||||
|
||||
await jobsRepo.updateJob(job.id, {
|
||||
suitabilityScore: score,
|
||||
suitabilityReason: reason,
|
||||
suitabilityAnalysis: analysis ? JSON.stringify(analysis) : undefined,
|
||||
sponsorMatchScore,
|
||||
sponsorMatchNames,
|
||||
...buildSponsorshipSignalsUpdate(job),
|
||||
...(shouldAutoSkip ? { status: "skipped" } : {}),
|
||||
});
|
||||
|
||||
if (shouldAutoSkip) {
|
||||
logger.info("Auto-skipped job due to low score", {
|
||||
} catch (error) {
|
||||
const message =
|
||||
error instanceof Error ? error.message : "Unknown scoring error";
|
||||
logger.error("Scoring task failed; continuing with remaining jobs", {
|
||||
jobId: job.id,
|
||||
title: job.title,
|
||||
score,
|
||||
threshold: autoSkipThreshold,
|
||||
error: message,
|
||||
});
|
||||
}
|
||||
|
||||
completed += 1;
|
||||
progressHelpers.scoringJob(completed, unprocessedJobs.length, job.title);
|
||||
scoredJobs.push({
|
||||
...job,
|
||||
suitabilityScore: score,
|
||||
suitabilityReason: reason,
|
||||
});
|
||||
},
|
||||
});
|
||||
|
||||
|
||||
@ -8,6 +8,7 @@ import { DEFAULT_JOB_OWNER_PROFILE_ID } from "@server/infra/job-owner-context";
|
||||
import {
|
||||
buildJobContentFingerprint,
|
||||
collectJobDedupKeys,
|
||||
PRIOR_SKIP_DISMISS_LOOKBACK_MS,
|
||||
} from "@shared/job-fingerprint";
|
||||
import { canonicalizeJobUrl } from "@shared/job-url-canonical";
|
||||
import type {
|
||||
@ -884,6 +885,180 @@ export async function deleteJobsByStatus(
|
||||
return result.changes;
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete discovered jobs that match skipped/applied ("archive") rows.
|
||||
*
|
||||
* Matching uses the same keys as import dismiss / auto-skip:
|
||||
* - employer+title content fingerprint against any skipped/applied row
|
||||
* - employer+title / employer+description keys for skips/applies in the
|
||||
* prior-skip lookback window
|
||||
*/
|
||||
export async function deleteDiscoveredJobsMatchingArchived(
|
||||
ownerProfileId: string = getJobOwnerProfileId() ??
|
||||
DEFAULT_JOB_OWNER_PROFILE_ID,
|
||||
): Promise<number> {
|
||||
const archivedRows = await db
|
||||
.select({
|
||||
id: jobs.id,
|
||||
employer: jobs.employer,
|
||||
title: jobs.title,
|
||||
jobDescription: jobs.jobDescription,
|
||||
contentFingerprint: jobs.contentFingerprint,
|
||||
updatedAt: jobs.updatedAt,
|
||||
})
|
||||
.from(jobs)
|
||||
.where(
|
||||
and(
|
||||
eq(jobs.ownerProfileId, ownerProfileId),
|
||||
inArray(jobs.status, ["skipped", "applied"]),
|
||||
),
|
||||
);
|
||||
|
||||
if (archivedRows.length === 0) return 0;
|
||||
|
||||
const archivedFingerprints = new Set<string>();
|
||||
const archivedDedupKeys = new Set<string>();
|
||||
const dismissCutoffMs = Date.now() - PRIOR_SKIP_DISMISS_LOOKBACK_MS;
|
||||
|
||||
for (const row of archivedRows) {
|
||||
const fingerprint =
|
||||
row.contentFingerprint?.trim() ||
|
||||
buildJobContentFingerprint({
|
||||
employer: row.employer,
|
||||
title: row.title,
|
||||
});
|
||||
if (fingerprint) {
|
||||
archivedFingerprints.add(fingerprint);
|
||||
}
|
||||
|
||||
const updatedMs = Date.parse(row.updatedAt);
|
||||
if (Number.isFinite(updatedMs) && updatedMs >= dismissCutoffMs) {
|
||||
for (const key of collectJobDedupKeys({
|
||||
employer: row.employer,
|
||||
title: row.title,
|
||||
jobDescription: row.jobDescription,
|
||||
})) {
|
||||
archivedDedupKeys.add(key);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
if (archivedFingerprints.size === 0 && archivedDedupKeys.size === 0) {
|
||||
return 0;
|
||||
}
|
||||
|
||||
const discoveredRows = await db
|
||||
.select({
|
||||
id: jobs.id,
|
||||
employer: jobs.employer,
|
||||
title: jobs.title,
|
||||
jobDescription: jobs.jobDescription,
|
||||
contentFingerprint: jobs.contentFingerprint,
|
||||
})
|
||||
.from(jobs)
|
||||
.where(
|
||||
and(
|
||||
eq(jobs.ownerProfileId, ownerProfileId),
|
||||
eq(jobs.status, "discovered"),
|
||||
),
|
||||
);
|
||||
|
||||
const idsToDelete: string[] = [];
|
||||
for (const row of discoveredRows) {
|
||||
const fingerprint =
|
||||
row.contentFingerprint?.trim() ||
|
||||
buildJobContentFingerprint({
|
||||
employer: row.employer,
|
||||
title: row.title,
|
||||
});
|
||||
if (fingerprint && archivedFingerprints.has(fingerprint)) {
|
||||
idsToDelete.push(row.id);
|
||||
continue;
|
||||
}
|
||||
if (archivedDedupKeys.size === 0) continue;
|
||||
const keys = collectJobDedupKeys({
|
||||
employer: row.employer,
|
||||
title: row.title,
|
||||
jobDescription: row.jobDescription,
|
||||
});
|
||||
if (keys.some((key) => archivedDedupKeys.has(key))) {
|
||||
idsToDelete.push(row.id);
|
||||
}
|
||||
}
|
||||
|
||||
if (idsToDelete.length === 0) return 0;
|
||||
|
||||
const result = await db
|
||||
.delete(jobs)
|
||||
.where(
|
||||
and(
|
||||
eq(jobs.ownerProfileId, ownerProfileId),
|
||||
inArray(jobs.id, idsToDelete),
|
||||
),
|
||||
)
|
||||
.run();
|
||||
return result.changes;
|
||||
}
|
||||
|
||||
/**
|
||||
* Probe discovered listing URLs and mark expired ones as `expired`.
|
||||
* Fail-open on network errors so flaky sites are not mass-expired.
|
||||
*/
|
||||
export async function expireDiscoveredJobsWithDeadListings(
|
||||
ownerProfileId: string = getJobOwnerProfileId() ??
|
||||
DEFAULT_JOB_OWNER_PROFILE_ID,
|
||||
options: {
|
||||
concurrency?: number;
|
||||
timeoutMs?: number;
|
||||
shouldStop?: () => boolean;
|
||||
} = {},
|
||||
): Promise<number> {
|
||||
const { filterExpiredJobListings } = await import(
|
||||
"@shared/job-listing-probe"
|
||||
);
|
||||
|
||||
const rows = await db
|
||||
.select({
|
||||
id: jobs.id,
|
||||
jobUrl: jobs.jobUrl,
|
||||
applicationLink: jobs.applicationLink,
|
||||
})
|
||||
.from(jobs)
|
||||
.where(
|
||||
and(
|
||||
eq(jobs.ownerProfileId, ownerProfileId),
|
||||
eq(jobs.status, "discovered"),
|
||||
),
|
||||
);
|
||||
|
||||
if (rows.length === 0) return 0;
|
||||
|
||||
const { kept } = await filterExpiredJobListings(rows, {
|
||||
concurrency: options.concurrency ?? 4,
|
||||
timeoutMs: options.timeoutMs ?? 5_000,
|
||||
shouldStop: options.shouldStop,
|
||||
});
|
||||
const keptIds = new Set(kept.map((row) => row.id));
|
||||
const expiredIds = rows
|
||||
.filter((row) => !keptIds.has(row.id))
|
||||
.map((row) => row.id);
|
||||
|
||||
if (expiredIds.length === 0) return 0;
|
||||
|
||||
const now = new Date().toISOString();
|
||||
const result = await db
|
||||
.update(jobs)
|
||||
.set({ status: "expired", updatedAt: now })
|
||||
.where(
|
||||
and(
|
||||
eq(jobs.ownerProfileId, ownerProfileId),
|
||||
inArray(jobs.id, expiredIds),
|
||||
),
|
||||
)
|
||||
.run();
|
||||
return result.changes;
|
||||
}
|
||||
|
||||
/**
|
||||
* Delete jobs with suitability score below threshold (excluding applied and in_progress jobs).
|
||||
*/
|
||||
|
||||
@ -1,5 +1,6 @@
|
||||
export * from "./extractors";
|
||||
export * from "./job-fingerprint";
|
||||
export * from "./job-listing-probe";
|
||||
export * from "./job-url-canonical";
|
||||
export * from "./keyword-match";
|
||||
export * from "./location-support";
|
||||
|
||||
123
shared/src/job-listing-probe.test.ts
Normal file
123
shared/src/job-listing-probe.test.ts
Normal file
@ -0,0 +1,123 @@
|
||||
import { describe, expect, it } from "vitest";
|
||||
import {
|
||||
filterExpiredJobListings,
|
||||
isGoogleJobsHost,
|
||||
listingLooksExpired,
|
||||
probeJobListingUrl,
|
||||
resolveJobListingProbeUrl,
|
||||
} from "./job-listing-probe.js";
|
||||
|
||||
describe("resolveJobListingProbeUrl", () => {
|
||||
it("prefers non-Google application links", () => {
|
||||
expect(
|
||||
resolveJobListingProbeUrl({
|
||||
jobUrl: "https://www.google.com/search?q=jobs",
|
||||
applicationLink: "https://bebee.com/ca/jobs/example",
|
||||
}),
|
||||
).toBe("https://bebee.com/ca/jobs/example");
|
||||
});
|
||||
|
||||
it("falls back to jobUrl when apply link is Google", () => {
|
||||
expect(
|
||||
resolveJobListingProbeUrl({
|
||||
jobUrl: "https://boards.greenhouse.io/acme/jobs/1",
|
||||
applicationLink: "https://www.google.com/search?q=jobs",
|
||||
}),
|
||||
).toBe("https://boards.greenhouse.io/acme/jobs/1");
|
||||
});
|
||||
});
|
||||
|
||||
describe("isGoogleJobsHost", () => {
|
||||
it("detects google hosts", () => {
|
||||
expect(isGoogleJobsHost("https://www.google.com/search?q=x")).toBe(true);
|
||||
expect(isGoogleJobsHost("https://www.google.ca/search?q=x")).toBe(true);
|
||||
expect(isGoogleJobsHost("https://bebee.com/jobs/1")).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("listingLooksExpired", () => {
|
||||
it("treats 410 and expired body text as expired", () => {
|
||||
expect(
|
||||
listingLooksExpired({
|
||||
status: 410,
|
||||
finalUrl: "https://bebee.com/jobs/1",
|
||||
body: "Job expired\nThis job listing is no longer available.",
|
||||
}),
|
||||
).toBe(true);
|
||||
});
|
||||
|
||||
it("keeps healthy 200 pages", () => {
|
||||
expect(
|
||||
listingLooksExpired({
|
||||
status: 200,
|
||||
finalUrl: "https://example.com/jobs/1",
|
||||
body: "<html><body>Apply now for Software Engineer</body></html>",
|
||||
}),
|
||||
).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe("probeJobListingUrl", () => {
|
||||
it("marks 410 responses expired", async () => {
|
||||
const fetchImpl = async () =>
|
||||
({
|
||||
url: "https://bebee.com/jobs/1",
|
||||
status: 410,
|
||||
text: async () => "Job expired",
|
||||
}) as Response;
|
||||
|
||||
const result = await probeJobListingUrl("https://bebee.com/jobs/1", {
|
||||
fetchImpl,
|
||||
});
|
||||
expect(result?.expired).toBe(true);
|
||||
expect(result?.status).toBe(410);
|
||||
});
|
||||
|
||||
it("returns null on network failure (fail-open)", async () => {
|
||||
const fetchImpl = async () => {
|
||||
throw new Error("network down");
|
||||
};
|
||||
const result = await probeJobListingUrl("https://example.com/jobs/1", {
|
||||
fetchImpl,
|
||||
});
|
||||
expect(result).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe("filterExpiredJobListings", () => {
|
||||
it("drops expired listings and keeps the rest", async () => {
|
||||
const fetchImpl = async (input: RequestInfo | URL) => {
|
||||
const url = String(input);
|
||||
if (url.includes("dead")) {
|
||||
return {
|
||||
url,
|
||||
status: 410,
|
||||
text: async () => "Job expired",
|
||||
} as Response;
|
||||
}
|
||||
return {
|
||||
url,
|
||||
status: 200,
|
||||
text: async () => "Apply now",
|
||||
} as Response;
|
||||
};
|
||||
|
||||
const { kept, dropped } = await filterExpiredJobListings(
|
||||
[
|
||||
{
|
||||
jobUrl: "https://example.com/alive",
|
||||
applicationLink: "https://example.com/alive",
|
||||
},
|
||||
{
|
||||
jobUrl: "https://example.com/dead",
|
||||
applicationLink: "https://example.com/dead",
|
||||
},
|
||||
],
|
||||
{ fetchImpl, concurrency: 2 },
|
||||
);
|
||||
|
||||
expect(dropped).toBe(1);
|
||||
expect(kept).toHaveLength(1);
|
||||
expect(kept[0]?.jobUrl).toContain("alive");
|
||||
});
|
||||
});
|
||||
172
shared/src/job-listing-probe.ts
Normal file
172
shared/src/job-listing-probe.ts
Normal file
@ -0,0 +1,172 @@
|
||||
/**
|
||||
* Lightweight job listing URL health check.
|
||||
*
|
||||
* Fail-open on network/timeout/5xx so flaky sites do not drop good jobs.
|
||||
* Hard-drop on 404/410 and common "expired / no longer available" page text.
|
||||
*/
|
||||
|
||||
export type JobListingProbeResult = {
|
||||
expired: boolean;
|
||||
status: number | null;
|
||||
finalUrl: string;
|
||||
};
|
||||
|
||||
const EXPIRED_URL_RE =
|
||||
/expired_jd_redirect|unavailable|no[_-]longer[_-]available|job[_-]expired|listing[_-]closed/i;
|
||||
|
||||
const EXPIRED_BODY_RE =
|
||||
/\bjob expired\b|\bthis job has expired\b|\bthis job listing is no longer available\b|\bno longer accepting applications\b|\bjob you were looking for is no longer available\b|\bthis job is no longer available\b|\bposition has been filled\b|\bjob posting is closed\b|\bthis posting has expired\b/i;
|
||||
|
||||
const GOOGLE_HOST_RE = /(?:^|\.)google\.[a-z.]+$/i;
|
||||
|
||||
export const DEFAULT_JOB_LISTING_PROBE_TIMEOUT_MS = 5_000;
|
||||
|
||||
export function isGoogleJobsHost(url: string): boolean {
|
||||
try {
|
||||
const host = new URL(url).hostname;
|
||||
return GOOGLE_HOST_RE.test(host);
|
||||
} catch {
|
||||
return false;
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Prefer the real apply URL over Google's share permalink (share pages often
|
||||
* stay 200 after the underlying listing dies).
|
||||
*/
|
||||
export function resolveJobListingProbeUrl(args: {
|
||||
jobUrl?: string | null;
|
||||
applicationLink?: string | null;
|
||||
}): string | null {
|
||||
const apply = args.applicationLink?.trim() || "";
|
||||
const jobUrl = args.jobUrl?.trim() || "";
|
||||
|
||||
if (apply && !isGoogleJobsHost(apply)) return apply;
|
||||
if (jobUrl && !isGoogleJobsHost(jobUrl)) return jobUrl;
|
||||
if (apply) return apply;
|
||||
return jobUrl || null;
|
||||
}
|
||||
|
||||
export function listingLooksExpired(args: {
|
||||
status: number;
|
||||
finalUrl: string;
|
||||
body: string;
|
||||
}): boolean {
|
||||
if (args.status === 404 || args.status === 410 || args.status === 451) {
|
||||
return true;
|
||||
}
|
||||
if (EXPIRED_URL_RE.test(args.finalUrl)) return true;
|
||||
// Only scan a prefix — expiry banners are near the top.
|
||||
const sample = args.body.slice(0, 12_000);
|
||||
return EXPIRED_BODY_RE.test(sample);
|
||||
}
|
||||
|
||||
async function fetchWithTimeout(
|
||||
url: string,
|
||||
timeoutMs: number,
|
||||
fetchImpl: typeof fetch,
|
||||
): Promise<Response> {
|
||||
const controller = new AbortController();
|
||||
const timer = setTimeout(() => controller.abort(), timeoutMs);
|
||||
try {
|
||||
return await fetchImpl(url, {
|
||||
method: "GET",
|
||||
redirect: "follow",
|
||||
signal: controller.signal,
|
||||
headers: {
|
||||
Accept: "text/html,application/xhtml+xml",
|
||||
"User-Agent": "JobOps/1.0 (+job-listing-probe)",
|
||||
},
|
||||
});
|
||||
} finally {
|
||||
clearTimeout(timer);
|
||||
}
|
||||
}
|
||||
|
||||
/**
|
||||
* Probe a listing URL. Returns `null` on transport failure (caller should keep).
|
||||
*/
|
||||
export async function probeJobListingUrl(
|
||||
url: string,
|
||||
options: {
|
||||
timeoutMs?: number;
|
||||
fetchImpl?: typeof fetch;
|
||||
} = {},
|
||||
): Promise<JobListingProbeResult | null> {
|
||||
const trimmed = url?.trim();
|
||||
if (!trimmed) return null;
|
||||
|
||||
const timeoutMs = options.timeoutMs ?? DEFAULT_JOB_LISTING_PROBE_TIMEOUT_MS;
|
||||
const fetchImpl = options.fetchImpl ?? fetch;
|
||||
|
||||
try {
|
||||
const response = await fetchWithTimeout(trimmed, timeoutMs, fetchImpl);
|
||||
const finalUrl = response.url || trimmed;
|
||||
// 5xx: fail-open
|
||||
if (response.status >= 500) {
|
||||
return { expired: false, status: response.status, finalUrl };
|
||||
}
|
||||
const body = await response.text();
|
||||
const expired = listingLooksExpired({
|
||||
status: response.status,
|
||||
finalUrl,
|
||||
body,
|
||||
});
|
||||
return { expired, status: response.status, finalUrl };
|
||||
} catch {
|
||||
return null;
|
||||
}
|
||||
}
|
||||
|
||||
export async function filterExpiredJobListings<
|
||||
T extends { jobUrl?: string | null; applicationLink?: string | null },
|
||||
>(
|
||||
jobs: readonly T[],
|
||||
options: {
|
||||
concurrency?: number;
|
||||
timeoutMs?: number;
|
||||
fetchImpl?: typeof fetch;
|
||||
shouldStop?: () => boolean;
|
||||
} = {},
|
||||
): Promise<{ kept: T[]; dropped: number }> {
|
||||
if (jobs.length === 0) return { kept: [], dropped: 0 };
|
||||
|
||||
const concurrency = Math.max(
|
||||
1,
|
||||
Math.min(6, Math.floor(options.concurrency ?? 4)),
|
||||
);
|
||||
const keptFlags = Array.from({ length: jobs.length }, () => true);
|
||||
let nextIndex = 0;
|
||||
let dropped = 0;
|
||||
|
||||
const worker = async () => {
|
||||
while (true) {
|
||||
if (options.shouldStop?.()) return;
|
||||
const index = nextIndex;
|
||||
nextIndex += 1;
|
||||
if (index >= jobs.length) return;
|
||||
|
||||
const job = jobs[index];
|
||||
const probeUrl = resolveJobListingProbeUrl(job);
|
||||
if (!probeUrl) continue;
|
||||
|
||||
const probe = await probeJobListingUrl(probeUrl, {
|
||||
timeoutMs: options.timeoutMs,
|
||||
fetchImpl: options.fetchImpl,
|
||||
});
|
||||
if (probe?.expired) {
|
||||
keptFlags[index] = false;
|
||||
dropped += 1;
|
||||
}
|
||||
}
|
||||
};
|
||||
|
||||
await Promise.all(
|
||||
Array.from({ length: Math.min(concurrency, jobs.length) }, () => worker()),
|
||||
);
|
||||
|
||||
return {
|
||||
kept: jobs.filter((_, index) => keptFlags[index]),
|
||||
dropped,
|
||||
};
|
||||
}
|
||||
Loading…
x
Reference in New Issue
Block a user