Humanize README tone #4

Merged
ilia merged 2 commits from docs/humanize-prose into master 2026-08-05 14:27:35 -05:00
+23 -21
View File
@@ -1,19 +1,21 @@
# Context Extractor
Status: active.
Capture console logs, network activity, JS errors, and clean markdown content
from a webpage, formatted as an AI-ready prompt. Ships three ways from one
shared core:
[![License: MIT](https://img.shields.io/badge/License-MIT-blue.svg)](LICENSE)
**AI-ready page context** console, network, errors, and clean markdown for agents and debugging.
Captures console, network, errors, and clean markdown for agents and debugging.
> Public mirror: [github.com/Gitilia/context-extractor](https://github.com/Gitilia/context-extractor)
Public mirror: [github.com/Gitilia/context-extractor](https://github.com/Gitilia/context-extractor).
- **`extension/`** — a browser extension (Manifest V3) for interactive use.
- **`automation/`** — a Python package for scripted/headless use with
- `extension/`: browser extension (Manifest V3) for interactive use.
- `automation/`: Python package for scripted/headless use with
Playwright or [Camoufox](https://camoufox.com).
- **`automation-js/`** — a Node/TS twin of `automation/` for Playwright
- `automation-js/`: Node/TS twin of `automation/` for Playwright
(`playwright-core`) or [camoufox-js](https://www.npmjs.com/package/camoufox-js)
automation that's already JS, so it can `import` the session directly
instead of shelling out to Python.
@@ -42,7 +44,7 @@ context-extractor/
```
`extension/core/dom.js` and `extension/core/prompt.js` are the single source
of truth for DOM/markdown/prompt logic the extension loads them directly as
of truth for DOM/markdown/prompt logic, the extension loads them directly as
content-script files, and **both** automation packages read the exact same
files (via symlinks) and run them with `page.evaluate()`. There's only one
place to fix a markdown-formatting bug, and the Python/JS test suites both
@@ -54,7 +56,7 @@ so a regression can't silently diverge between the two.
The extension (`extension/manifest.json`), the Python package
(`automation/pyproject.toml`), and the Node package
(`automation-js/package.json`) share **one product version** and are bumped
together they ship the same `core/*.js`, so a version like `1.4.0` means
together, they ship the same `core/*.js`, so a version like `1.4.0` means
the same extraction behavior in all three.
## Use from another Cursor project
@@ -82,7 +84,7 @@ pytest -q
## 1. Browser extension (Chrome and Brave)
Brave is Chromium under the hood and loads unpacked Manifest V3 extensions
exactly like Chrome does there is no separate "Brave build." Same steps in
exactly like Chrome does, there is no separate "Brave build." Same steps in
both:
1. Go to `chrome://extensions` (or `brave://extensions`).
@@ -100,7 +102,7 @@ element), and **Copy** to put a ready-to-paste prompt on the clipboard.
The original console/fetch/XHR patcher was injected as an inline `<script>`
tag, which strict-CSP sites (`script-src` without `'unsafe-inline'`) silently
block. This version declares `page-patcher.js` as a second content script
with `"world": "MAIN"` (Chrome/Brave 111+) instead it runs directly in the
with `"world": "MAIN"` (Chrome/Brave 111+) instead, it runs directly in the
page's own JS realm without ever going through the page's HTML parser, so
it isn't subject to that CSP restriction. It talks back to the isolated-world
`content.js` (which owns the store, the element picker, and
@@ -113,7 +115,7 @@ ISOLATED worlds don't share JS globals.
python3 scripts/package_extension.py # -> dist/context-extractor-extension.zip
```
Not needed for local use "Load unpacked" against `extension/` is enough.
Not needed for local use, "Load unpacked" against `extension/` is enough.
## 2. Playwright / Camoufox automation
@@ -136,7 +138,7 @@ with sync_playwright() as p:
print(session.build_ai_prompt()) # or session.extract_markdown("#main")
```
Camoufox (sync or async) works the same way see
Camoufox (sync or async) works the same way, see
`automation/examples/scrape_camoufox.py` and
`automation/examples/scrape_camoufox_async.py`. Use `AsyncExtractorSession`
for `camoufox.async_api.AsyncCamoufox` / `playwright.async_api`.
@@ -147,7 +149,7 @@ Headless CLI, handy in CI:
context-extractor https://example.com --selector "#main" --engine camoufox --out prompt.md
```
### JS/TS twin (`automation-js/`) for Node/Playwright or camoufox-js automation
### JS/TS twin (`automation-js/`), for Node/Playwright or camoufox-js automation
Same idea, no Python subprocess:
@@ -172,36 +174,36 @@ See `automation-js/README.md`.
Camoufox executes Playwright's own JS (`page.add_init_script()`,
`page.evaluate()` writes) in an isolated context by design, as an
anti-fingerprinting measure this is a
anti-fingerprinting measure, this is a
[known, documented limitation](https://github.com/daijro/camoufox/issues/48)
that breaks naive init-script injection. Rather than fight that, the
automation layer captures console/network/errors via Playwright's *native*
event hooks (`page.on("console"/"request"/"requestfinished"/"requestfailed"/"pageerror")`)
instead of injecting JS into the page at all. This is more robust than the
extension's approach (it can't be blocked by page CSP, and it's not limited
to fetch/XHR it sees every resource type), and it works identically
to fetch/XHR, it sees every resource type), and it works identically
whether the page is Chromium, Firefox, WebKit, or Camoufox.
Markdown extraction and CSS-selector logic *do* run inside the page (they
need real DOM traversal), via `page.evaluate()` against `core/dom.js`. That's
safe under Camoufox's default isolated world because that code only *reads*
the DOM and mutates a detached clone it never writes to the live page —
the DOM and mutates a detached clone, it never writes to the live page —
so no `main_world_eval` workaround is needed there either.
## Extraction behavior worth knowing
- `extractMarkdown()` walks the *rendered* page and skips anything actually
invisible: `display:none`, `visibility:hidden`, `[hidden]`,
`aria-hidden="true"` however it's hidden (inline style, stylesheet class,
`aria-hidden="true"`, however it's hidden (inline style, stylesheet class,
or attribute). SPAs routinely stash large hydration/experiment JSON
payloads in hidden DOM nodes, not just `<script>` tags (LinkedIn is a good
example its DOM without this filter is 700k+ characters, almost all of
example, its DOM without this filter is 700k+ characters, almost all of
it invisible config junk). Without this check you'd be feeding an LLM
chameleon experiment payloads instead of page content.
- `build_ai_prompt()` / the extension **Copy** button cap page content at **20,000 chars**
by default (`maxChars` in JS, `max_chars=`/`--max-chars` in Python/CLI). A
full `<body>` extraction on a JS-heavy SPA can still be enormous even after
hidden-node filtering the cap is a safety net, not a summarizer. Prefer
hidden-node filtering, the cap is a safety net, not a summarizer. Prefer
a narrower selector (**Pick** in the extension, or a known selector in
automation) over relying on the cap for real content.
@@ -216,13 +218,13 @@ so no `main_world_eval` workaround is needed there either.
uses Playwright's own request events.
- `page-patcher.js` still can't see activity from a page's Web Workers or
Service Workers; neither can the automation layer without additional
`page.on("worker")` wiring (not implemented open an issue/extend
`page.on("worker")` wiring (not implemented, open an issue/extend
`session.py` if you need it).
- Hidden-node filtering catches `display:none`/`visibility:hidden`/`[hidden]`/
`aria-hidden`, but not "visually clipped but AT-readable" sr-only patterns
(e.g. `position:absolute;clip:rect(0,0,0,0)`) those are left in on
(e.g. `position:absolute;clip:rect(0,0,0,0)`), those are left in on
purpose since they're usually real text, not hydration data.
## License
MIT see [LICENSE](LICENSE).
MIT, see [LICENSE](LICENSE).