Context Extractor

Status: active.

Capture console logs, network activity, JS errors, and clean markdown content from a webpage, formatted as an AI-ready prompt. Ships three ways from one shared core:

Captures console, network, errors, and clean markdown for agents and debugging.

Public mirror: github.com/Gitilia/context-extractor.

• extension/: browser extension (Manifest V3) for interactive use.
• automation/: Python package for scripted/headless use with Playwright or Camoufox.
• automation-js/: Node/TS twin of automation/ for Playwright (playwright-core) or camoufox-js automation that's already JS, so it can import the session directly instead of shelling out to Python.

context-extractor/
├── extension/                   browser extension (Chrome + Brave, unpacked)
│   ├── core/dom.js              ← shared: selector + markdown extraction
│   ├── core/prompt.js           ← shared: AI-prompt formatter
│   ├── page-patcher.js          MAIN-world console/fetch/XHR/error capture
│   ├── content.js               ISOLATED-world store, picker, messaging
│   ├── background.js
│   └── popup.html / popup.js
├── automation/                  Python package for Playwright / Camoufox
│   ├── context_extractor/       session + CLI
│   │   └── js/                  symlinks → extension/core/*.js
│   └── tests/                   pytest + fixture page
├── automation-js/                Node/TS package for Playwright / camoufox-js
│   ├── src/session.ts           ← ExtractorSession (async, one class)
│   │   └── js/                  symlinks → extension/core/*.js
│   └── tests/                   vitest + real Chromium, shared fixture
├── scripts/package_extension.py optional zip for distribution
├── .gitea/workflows/ci.yml      Gitea Actions: JS lint + pytest + package smoke
├── CURSOR_PROMPT.md             paste into other Cursor chats
└── Makefile                     make test / make lint / make package

extension/core/dom.js and extension/core/prompt.js are the single source of truth for DOM/markdown/prompt logic, the extension loads them directly as content-script files, and both automation packages read the exact same files (via symlinks) and run them with page.evaluate(). There's only one place to fix a markdown-formatting bug, and the Python/JS test suites both run against the same fixture page (automation/tests/fixtures/sample.html) so a regression can't silently diverge between the two.

Versioning

The extension (extension/manifest.json), the Python package (automation/pyproject.toml), and the Node package (automation-js/package.json) share one product version and are bumped together, they ship the same core/*.js, so a version like 1.4.0 means the same extraction behavior in all three.

Use from another Cursor project

Copy the block in CURSOR_PROMPT.md into that project's chat (or drop it in .cursor/rules/).

Tests

make install   # once
make ci        # lint + pytest + vitest + package smoke (same checks as Gitea Actions)

Or manually:

cd automation
python3 -m venv .venv && source .venv/bin/activate
pip install -e ".[dev]"
playwright install chromium
pytest -q

1. Browser extension (Chrome and Brave)

Brave is Chromium under the hood and loads unpacked Manifest V3 extensions exactly like Chrome does, there is no separate "Brave build." Same steps in both:

1. Go to chrome://extensions (or brave://extensions).
2. Enable Developer Mode.
3. Load unpacked → select the extension/ folder.
4. Reload any tab you want to capture from (content scripts only attach to pages loaded/reloaded after install).

Click the toolbar icon to see captured console/network/errors, extract markdown from the whole page or a CSS selector (use Pick to click an element), and Copy to put a ready-to-paste prompt on the clipboard.

What changed vs. a typical extension implementation

The original console/fetch/XHR patcher was injected as an inline <script> tag, which strict-CSP sites (script-src without 'unsafe-inline') silently block. This version declares page-patcher.js as a second content script with "world": "MAIN" (Chrome/Brave 111+) instead, it runs directly in the page's own JS realm without ever going through the page's HTML parser, so it isn't subject to that CSP restriction. It talks back to the isolated-world content.js (which owns the store, the element picker, and chrome.runtime messaging) via a CustomEvent bridge, since MAIN and ISOLATED worlds don't share JS globals.

Packaging a zip (optional)

python3 scripts/package_extension.py   # -> dist/context-extractor-extension.zip

Not needed for local use, "Load unpacked" against extension/ is enough.

2. Playwright / Camoufox automation

cd automation
python3 -m venv .venv && source .venv/bin/activate
pip install -e .                 # plain Playwright
pip install -e ".[camoufox]"     # + Camoufox extra
playwright install chromium      # only needed for the plain-Playwright path

from playwright.sync_api import sync_playwright
from context_extractor import ExtractorSession

with sync_playwright() as p:
    page = p.chromium.launch().new_page()
    session = ExtractorSession(page)   # attach BEFORE navigating
    page.goto("https://example.com", wait_until="networkidle")
    print(session.build_ai_prompt())    # or session.extract_markdown("#main")

Camoufox (sync or async) works the same way, see automation/examples/scrape_camoufox.py and automation/examples/scrape_camoufox_async.py. Use AsyncExtractorSession for camoufox.async_api.AsyncCamoufox / playwright.async_api.

Headless CLI, handy in CI:

context-extractor https://example.com --selector "#main" --engine camoufox --out prompt.md

JS/TS twin (automation-js/), for Node/Playwright or camoufox-js automation

Same idea, no Python subprocess:

cd automation-js
npm install && npm run build

import { chromium } from 'playwright-core';
import { ExtractorSession } from 'context-extractor'; // automation-js

const page = await (await chromium.launch()).newPage();
const session = new ExtractorSession(page); // attach BEFORE navigating
await page.goto('https://example.com', { waitUntil: 'networkidle' });
console.log(await session.buildAiPrompt()); // or session.extractMarkdown('#main')

See automation-js/README.md.

Why automation doesn't reuse the extension's patcher

Camoufox executes Playwright's own JS (page.add_init_script(), page.evaluate() writes) in an isolated context by design, as an anti-fingerprinting measure, this is a known, documented limitation that breaks naive init-script injection. Rather than fight that, the automation layer captures console/network/errors via Playwright's native event hooks (page.on("console"/"request"/"requestfinished"/"requestfailed"/"pageerror")) instead of injecting JS into the page at all. This is more robust than the extension's approach (it can't be blocked by page CSP, and it's not limited to fetch/XHR, it sees every resource type), and it works identically whether the page is Chromium, Firefox, WebKit, or Camoufox.

Markdown extraction and CSS-selector logic do run inside the page (they need real DOM traversal), via page.evaluate() against core/dom.js. That's safe under Camoufox's default isolated world because that code only reads the DOM and mutates a detached clone, it never writes to the live page — so no main_world_eval workaround is needed there either.

Extraction behavior worth knowing

• extractMarkdown() walks the rendered page and skips anything actually invisible: display:none, visibility:hidden, [hidden], aria-hidden="true", however it's hidden (inline style, stylesheet class, or attribute). SPAs routinely stash large hydration/experiment JSON payloads in hidden DOM nodes, not just <script> tags (LinkedIn is a good example, its DOM without this filter is 700k+ characters, almost all of it invisible config junk). Without this check you'd be feeding an LLM chameleon experiment payloads instead of page content.
• Long data: URLs (e.g. inlined images) are truncated in markdown links and in AI-prompt network lines via truncateDataUri, so dumps stay usable.
• inventoryControls() / inventory_controls() lists buttons, menuitems, and aria-label nodes with a visible vs hidden flag (capped). buildAiPrompt includes an Interesting Controls section so agents can spot hidden Radix twins without dumping the whole DOM.
• build_ai_prompt() / the extension Copy button cap page content at 20,000 chars by default (maxChars in JS, max_chars=/--max-chars in Python/CLI). A full <body> extraction on a JS-heavy SPA can still be enormous even after hidden-node filtering, the cap is a safety net, not a summarizer. Prefer a narrower selector (Pick in the extension, or a known selector in automation) over relying on the cap for real content.

Known limitations

• The element-picker UI (click-to-grab-a-selector) is extension-only; it's a visual affordance that doesn't make sense in headless automation. For scripted use, pass the selector you already know to extract_markdown() / build_ai_prompt().
• The extension's network capture only sees fetch + XMLHttpRequest (it's patched in JS). The automation layer sees every resource type since it uses Playwright's own request events.
• page-patcher.js still can't see activity from a page's Web Workers or Service Workers; neither can the automation layer without additional page.on("worker") wiring (not implemented, open an issue/extend session.py if you need it).
• Hidden-node filtering catches display:none/visibility:hidden/[hidden]/ aria-hidden, but not "visually clipped but AT-readable" sr-only patterns (e.g. position:absolute;clip:rect(0,0,0,0)), those are left in on purpose since they're usually real text, not hydration data.

License

MIT, see LICENSE.