Merge pull request 'Humanize README tone' (#4) from docs/humanize-prose into master
This commit was merged in pull request #4.
This commit is contained in:
@@ -1,19 +1,21 @@
|
||||
# Context Extractor
|
||||
|
||||
Status: active.
|
||||
|
||||
Capture console logs, network activity, JS errors, and clean markdown content
|
||||
from a webpage, formatted as an AI-ready prompt. Ships three ways from one
|
||||
shared core:
|
||||
|
||||
[](LICENSE)
|
||||
|
||||
**AI-ready page context** — console, network, errors, and clean markdown for agents and debugging.
|
||||
Captures console, network, errors, and clean markdown for agents and debugging.
|
||||
|
||||
> Public mirror: [github.com/Gitilia/context-extractor](https://github.com/Gitilia/context-extractor)
|
||||
Public mirror: [github.com/Gitilia/context-extractor](https://github.com/Gitilia/context-extractor).
|
||||
|
||||
- **`extension/`** — a browser extension (Manifest V3) for interactive use.
|
||||
- **`automation/`** — a Python package for scripted/headless use with
|
||||
- `extension/`: browser extension (Manifest V3) for interactive use.
|
||||
- `automation/`: Python package for scripted/headless use with
|
||||
Playwright or [Camoufox](https://camoufox.com).
|
||||
- **`automation-js/`** — a Node/TS twin of `automation/` for Playwright
|
||||
- `automation-js/`: Node/TS twin of `automation/` for Playwright
|
||||
(`playwright-core`) or [camoufox-js](https://www.npmjs.com/package/camoufox-js)
|
||||
automation that's already JS, so it can `import` the session directly
|
||||
instead of shelling out to Python.
|
||||
@@ -42,7 +44,7 @@ context-extractor/
|
||||
```
|
||||
|
||||
`extension/core/dom.js` and `extension/core/prompt.js` are the single source
|
||||
of truth for DOM/markdown/prompt logic — the extension loads them directly as
|
||||
of truth for DOM/markdown/prompt logic, the extension loads them directly as
|
||||
content-script files, and **both** automation packages read the exact same
|
||||
files (via symlinks) and run them with `page.evaluate()`. There's only one
|
||||
place to fix a markdown-formatting bug, and the Python/JS test suites both
|
||||
@@ -54,7 +56,7 @@ so a regression can't silently diverge between the two.
|
||||
The extension (`extension/manifest.json`), the Python package
|
||||
(`automation/pyproject.toml`), and the Node package
|
||||
(`automation-js/package.json`) share **one product version** and are bumped
|
||||
together — they ship the same `core/*.js`, so a version like `1.4.0` means
|
||||
together, they ship the same `core/*.js`, so a version like `1.4.0` means
|
||||
the same extraction behavior in all three.
|
||||
|
||||
## Use from another Cursor project
|
||||
@@ -82,7 +84,7 @@ pytest -q
|
||||
## 1. Browser extension (Chrome and Brave)
|
||||
|
||||
Brave is Chromium under the hood and loads unpacked Manifest V3 extensions
|
||||
exactly like Chrome does — there is no separate "Brave build." Same steps in
|
||||
exactly like Chrome does, there is no separate "Brave build." Same steps in
|
||||
both:
|
||||
|
||||
1. Go to `chrome://extensions` (or `brave://extensions`).
|
||||
@@ -100,7 +102,7 @@ element), and **Copy** to put a ready-to-paste prompt on the clipboard.
|
||||
The original console/fetch/XHR patcher was injected as an inline `<script>`
|
||||
tag, which strict-CSP sites (`script-src` without `'unsafe-inline'`) silently
|
||||
block. This version declares `page-patcher.js` as a second content script
|
||||
with `"world": "MAIN"` (Chrome/Brave 111+) instead — it runs directly in the
|
||||
with `"world": "MAIN"` (Chrome/Brave 111+) instead, it runs directly in the
|
||||
page's own JS realm without ever going through the page's HTML parser, so
|
||||
it isn't subject to that CSP restriction. It talks back to the isolated-world
|
||||
`content.js` (which owns the store, the element picker, and
|
||||
@@ -113,7 +115,7 @@ ISOLATED worlds don't share JS globals.
|
||||
python3 scripts/package_extension.py # -> dist/context-extractor-extension.zip
|
||||
```
|
||||
|
||||
Not needed for local use — "Load unpacked" against `extension/` is enough.
|
||||
Not needed for local use, "Load unpacked" against `extension/` is enough.
|
||||
|
||||
## 2. Playwright / Camoufox automation
|
||||
|
||||
@@ -136,7 +138,7 @@ with sync_playwright() as p:
|
||||
print(session.build_ai_prompt()) # or session.extract_markdown("#main")
|
||||
```
|
||||
|
||||
Camoufox (sync or async) works the same way — see
|
||||
Camoufox (sync or async) works the same way, see
|
||||
`automation/examples/scrape_camoufox.py` and
|
||||
`automation/examples/scrape_camoufox_async.py`. Use `AsyncExtractorSession`
|
||||
for `camoufox.async_api.AsyncCamoufox` / `playwright.async_api`.
|
||||
@@ -147,7 +149,7 @@ Headless CLI, handy in CI:
|
||||
context-extractor https://example.com --selector "#main" --engine camoufox --out prompt.md
|
||||
```
|
||||
|
||||
### JS/TS twin (`automation-js/`) — for Node/Playwright or camoufox-js automation
|
||||
### JS/TS twin (`automation-js/`), for Node/Playwright or camoufox-js automation
|
||||
|
||||
Same idea, no Python subprocess:
|
||||
|
||||
@@ -172,36 +174,36 @@ See `automation-js/README.md`.
|
||||
|
||||
Camoufox executes Playwright's own JS (`page.add_init_script()`,
|
||||
`page.evaluate()` writes) in an isolated context by design, as an
|
||||
anti-fingerprinting measure — this is a
|
||||
anti-fingerprinting measure, this is a
|
||||
[known, documented limitation](https://github.com/daijro/camoufox/issues/48)
|
||||
that breaks naive init-script injection. Rather than fight that, the
|
||||
automation layer captures console/network/errors via Playwright's *native*
|
||||
event hooks (`page.on("console"/"request"/"requestfinished"/"requestfailed"/"pageerror")`)
|
||||
instead of injecting JS into the page at all. This is more robust than the
|
||||
extension's approach (it can't be blocked by page CSP, and it's not limited
|
||||
to fetch/XHR — it sees every resource type), and it works identically
|
||||
to fetch/XHR, it sees every resource type), and it works identically
|
||||
whether the page is Chromium, Firefox, WebKit, or Camoufox.
|
||||
|
||||
Markdown extraction and CSS-selector logic *do* run inside the page (they
|
||||
need real DOM traversal), via `page.evaluate()` against `core/dom.js`. That's
|
||||
safe under Camoufox's default isolated world because that code only *reads*
|
||||
the DOM and mutates a detached clone — it never writes to the live page —
|
||||
the DOM and mutates a detached clone, it never writes to the live page —
|
||||
so no `main_world_eval` workaround is needed there either.
|
||||
|
||||
## Extraction behavior worth knowing
|
||||
|
||||
- `extractMarkdown()` walks the *rendered* page and skips anything actually
|
||||
invisible: `display:none`, `visibility:hidden`, `[hidden]`,
|
||||
`aria-hidden="true"` — however it's hidden (inline style, stylesheet class,
|
||||
`aria-hidden="true"`, however it's hidden (inline style, stylesheet class,
|
||||
or attribute). SPAs routinely stash large hydration/experiment JSON
|
||||
payloads in hidden DOM nodes, not just `<script>` tags (LinkedIn is a good
|
||||
example — its DOM without this filter is 700k+ characters, almost all of
|
||||
example, its DOM without this filter is 700k+ characters, almost all of
|
||||
it invisible config junk). Without this check you'd be feeding an LLM
|
||||
chameleon experiment payloads instead of page content.
|
||||
- `build_ai_prompt()` / the extension **Copy** button cap page content at **20,000 chars**
|
||||
by default (`maxChars` in JS, `max_chars=`/`--max-chars` in Python/CLI). A
|
||||
full `<body>` extraction on a JS-heavy SPA can still be enormous even after
|
||||
hidden-node filtering — the cap is a safety net, not a summarizer. Prefer
|
||||
hidden-node filtering, the cap is a safety net, not a summarizer. Prefer
|
||||
a narrower selector (**Pick** in the extension, or a known selector in
|
||||
automation) over relying on the cap for real content.
|
||||
|
||||
@@ -216,13 +218,13 @@ so no `main_world_eval` workaround is needed there either.
|
||||
uses Playwright's own request events.
|
||||
- `page-patcher.js` still can't see activity from a page's Web Workers or
|
||||
Service Workers; neither can the automation layer without additional
|
||||
`page.on("worker")` wiring (not implemented — open an issue/extend
|
||||
`page.on("worker")` wiring (not implemented, open an issue/extend
|
||||
`session.py` if you need it).
|
||||
- Hidden-node filtering catches `display:none`/`visibility:hidden`/`[hidden]`/
|
||||
`aria-hidden`, but not "visually clipped but AT-readable" sr-only patterns
|
||||
(e.g. `position:absolute;clip:rect(0,0,0,0)`) — those are left in on
|
||||
(e.g. `position:absolute;clip:rect(0,0,0,0)`), those are left in on
|
||||
purpose since they're usually real text, not hydration data.
|
||||
|
||||
## License
|
||||
|
||||
MIT — see [LICENSE](LICENSE).
|
||||
MIT, see [LICENSE](LICENSE).
|
||||
|
||||
Reference in New Issue
Block a user