Initial scaffold: capture + vision modules with detect CLI

- capture/: frame sampling at configurable fps, ffmpeg audio extraction (WAV, whisper-ready)
- vision/: changed-pixel motion scoring, stable/moving segmentation, contour + perspective document detection, Laplacian sharpness scoring, optional CLAHE enhancement
- pipeline: two-pass detect (motion timeline, then best-frame crop per stable window) writing crops, debug frames, report.json, and motion_scores.csv
- CLI: probe / detect / extract-audio subcommands
- config.yaml with tunable thresholds; placeholder packages for events, transcribe, ocr, naming, pdf, review_cli
- synthetic sample video generator + 16 unit tests

.gitignore

@@ -0,0 +1,16 @@
+__pycache__/
+*.py[cod]
+.venv/
+venv/
+.pytest_cache/
+.mypy_cache/
+*.egg-info/
+
+# Runtime artifacts
+output/
+sample_data/videos/*.mp4
+sample_data/videos/*.mov
+!sample_data/videos/.gitkeep
+*.wav
+
+.DS_Store

README.md

@@ -0,0 +1,79 @@
+# PaperPod
+
+Local, privacy-first tool that converts a single overhead video recording of
+documents (receipts, letters, multi-page stacks) into individual, properly
+named PDFs ready for a Paperless-ngx consume folder. All processing runs
+offline on your machine.
+
+## How it works
+
+1. Record one continuous overhead video, placing documents under the camera
+   one at a time (optionally saying out loud what each one is).
+2. PaperPod finds "stable windows" where nothing is moving, picks the sharpest
+   frame in each, detects the document outline, and produces a
+   perspective-corrected crop.
+3. (Upcoming) Spoken descriptions (faster-whisper) or OCR (Tesseract) name
+   each document; multi-page stacks are grouped into single PDFs; a review
+   step lets you rename/merge/split before export.
+
+## Status
+
+| Module | Purpose | Status |
+|---|---|---|
+| `capture/` | Frame sampling + audio extraction from video files | Built |
+| `vision/` | Motion detection, document contours, perspective crop, sharpness | Built |
+| `events/` | State machine: placed / page-flipped / cleared, pod grouping | Planned |
+| `transcribe/` | Local speech-to-text (faster-whisper) | Planned |
+| `ocr/` | OCR fallback naming (Tesseract) | Planned |
+| `naming/` | Final filename assembly + summary CSV | Planned |
+| `pdf/` | PDF assembly + Paperless-ngx consume staging | Planned |
+| `review_cli/` | Pre-export review (rename/merge/split) | Planned |
+
+## Setup
+
+Requires Python 3.11+ and ffmpeg (`brew install ffmpeg`).
+
+```bash
+python3 -m venv .venv
+source .venv/bin/activate
+pip install -r requirements.txt
+```
+
+## Usage
+
+```bash
+# Generate a synthetic test video (no camera needed)
+python sample_data/make_sample_video.py
+
+# Inspect a video
+python -m paperpod probe sample_data/videos/synthetic_sample.mp4
+
+# Detect stable windows + document candidates
+python -m paperpod detect sample_data/videos/synthetic_sample.mp4
+
+# Extract the audio track (16 kHz mono WAV, whisper-ready)
+python -m paperpod extract-audio my_recording.mp4
+```
+
+`detect` writes to `output/<video-name>/`:
+
+- `crops/window_NNN.png` — perspective-corrected document candidates
+- `frames/window_NNN_full.png` — the full best frame per window (debugging)
+- `report.json` — video metadata, motion events, stable windows, detections
+- `motion_scores.csv` — per-sample motion scores, for tuning `motion.threshold`
+
+## Tuning
+
+All thresholds live in `config.yaml` (motion sensitivity, stable window
+duration, contour size floor, Canny thresholds, speech matching window,
+output paths). Plot `motion_scores.csv` to pick a `motion.threshold` that
+separates your camera's noise floor from real hand movement.
+
+## Tests
+
+```bash
+python -m pytest
+```
+
+Tests are self-contained: they synthesize frames and tiny videos on the fly,
+no sample assets required.

config.yaml

@@ -0,0 +1,44 @@
+# PaperPod tunable thresholds. All values here override the built-in defaults
+# in paperpod/config.py; delete a key to fall back to the default.
+
+capture:
+  # Frames per second to sample from the source video for analysis.
+  sample_fps: 8
+  # Frames are downscaled to this width for motion analysis (full resolution
+  # is still used for the final document crops).
+  processing_width: 960
+
+audio:
+  # WAV sample rate for the extracted audio track (16 kHz mono = whisper-ready).
+  sample_rate: 16000
+
+motion:
+  # Gaussian blur kernel applied before frame differencing (odd number).
+  blur_ksize: 21
+  # A pixel counts as "changed" if its gray level moved by more than this (0-255).
+  pixel_threshold: 12
+  # Fraction of changed pixels (0.0-1.0) above which a frame counts as "moving".
+  # Raise if sensor noise triggers false motion; lower if small gestures
+  # (page flips) are being missed.
+  threshold: 0.02
+  # A stable stretch must last at least this long (seconds) to count as a window.
+  stable_min_duration_s: 1.0
+
+document:
+  # Contours smaller than this fraction of the frame area are ignored.
+  min_area_ratio: 0.04
+  # Canny edge detection thresholds.
+  canny_low: 50
+  canny_high: 150
+  # Apply contrast enhancement (CLAHE) to crops for OCR-readiness.
+  enhance: false
+
+speech:
+  # Window around a capture event in which to look for a spoken description.
+  window_before_s: 3.0
+  window_after_s: 5.0
+
+output:
+  dir: ./output
+  # Paperless-ngx consume directory (module 7, not wired up yet).
+  consume_dir: null

conftest.py

@@ -0,0 +1 @@
+# Ensures the repo root is on sys.path so tests can import `paperpod`.

paperpod/__init__.py

@@ -0,0 +1,3 @@
+"""PaperPod: turn one overhead video into individual, named PDF documents."""
+
+__version__ = "0.1.0"

paperpod/__main__.py

@@ -0,0 +1,4 @@
+from paperpod.cli import main
+
+if __name__ == "__main__":
+    raise SystemExit(main())

paperpod/capture/__init__.py

@@ -0,0 +1,6 @@
+"""Module 1: video/audio ingestion (pre-recorded files for now)."""
+
+from paperpod.capture.audio import extract_audio
+from paperpod.capture.video import Frame, VideoMeta, iter_frames, probe_video
+
+__all__ = ["Frame", "VideoMeta", "iter_frames", "probe_video", "extract_audio"]

paperpod/capture/audio.py

@@ -0,0 +1,40 @@
+"""Audio track extraction (WAV, whisper-ready) via ffmpeg."""
+
+from __future__ import annotations
+
+import shutil
+import subprocess
+from pathlib import Path
+
+
+def extract_audio(
+    video_path: str | Path,
+    wav_path: str | Path,
+    sample_rate: int = 16000,
+) -> Path:
+    """Extract the audio track as mono WAV at sample_rate. Requires ffmpeg."""
+    video_path = Path(video_path)
+    wav_path = Path(wav_path)
+    if not video_path.exists():
+        raise FileNotFoundError(f"Video not found: {video_path}")
+    if shutil.which("ffmpeg") is None:
+        raise RuntimeError(
+            "ffmpeg not found on PATH. Install it with: brew install ffmpeg"
+        )
+    wav_path.parent.mkdir(parents=True, exist_ok=True)
+    cmd = [
+        "ffmpeg",
+        "-y",
+        "-i", str(video_path),
+        "-vn",
+        "-ac", "1",
+        "-ar", str(sample_rate),
+        "-f", "wav",
+        str(wav_path),
+    ]
+    result = subprocess.run(cmd, capture_output=True, text=True)
+    if result.returncode != 0:
+        stderr = result.stderr.strip().splitlines()
+        tail = "\n".join(stderr[-5:])
+        raise RuntimeError(f"ffmpeg failed extracting audio:\n{tail}")
+    return wav_path

paperpod/capture/video.py

@@ -0,0 +1,91 @@
+"""Frame extraction from a pre-recorded video file."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+from pathlib import Path
+from typing import Iterator
+
+import cv2
+import numpy as np
+
+
+@dataclass
+class VideoMeta:
+    path: str
+    fps: float
+    frame_count: int
+    duration_s: float
+    width: int
+    height: int
+
+
+@dataclass
+class Frame:
+    index: int  # frame index in the source video (native fps)
+    t: float  # timestamp in seconds
+    image: np.ndarray  # BGR
+
+
+def probe_video(path: str | Path) -> VideoMeta:
+    path = Path(path)
+    if not path.exists():
+        raise FileNotFoundError(f"Video not found: {path}")
+    cap = cv2.VideoCapture(str(path))
+    if not cap.isOpened():
+        raise ValueError(f"Could not open video (unsupported codec?): {path}")
+    try:
+        fps = cap.get(cv2.CAP_PROP_FPS) or 0.0
+        frame_count = int(cap.get(cv2.CAP_PROP_FRAME_COUNT))
+        width = int(cap.get(cv2.CAP_PROP_FRAME_WIDTH))
+        height = int(cap.get(cv2.CAP_PROP_FRAME_HEIGHT))
+    finally:
+        cap.release()
+    if fps <= 0:
+        raise ValueError(f"Video reports invalid fps ({fps}): {path}")
+    return VideoMeta(
+        path=str(path),
+        fps=fps,
+        frame_count=frame_count,
+        duration_s=frame_count / fps if frame_count > 0 else 0.0,
+        width=width,
+        height=height,
+    )
+
+
+def iter_frames(
+    path: str | Path,
+    sample_fps: float = 8.0,
+    resize_width: int | None = None,
+) -> Iterator[Frame]:
+    """Sequentially decode the video, yielding frames at ~sample_fps.
+
+    Decoding is sequential (no seeking) so it works reliably across codecs.
+    If resize_width is set, frames are downscaled to that width preserving
+    aspect ratio (used for motion analysis; crops use full resolution).
+    """
+    meta = probe_video(path)
+    step = max(1, round(meta.fps / sample_fps))
+
+    cap = cv2.VideoCapture(str(path))
+    try:
+        index = 0
+        while True:
+            ok = cap.grab()
+            if not ok:
+                break
+            if index % step == 0:
+                ok, image = cap.retrieve()
+                if not ok:
+                    break
+                if resize_width is not None and image.shape[1] > resize_width:
+                    scale = resize_width / image.shape[1]
+                    image = cv2.resize(
+                        image,
+                        (resize_width, max(1, round(image.shape[0] * scale))),
+                        interpolation=cv2.INTER_AREA,
+                    )
+                yield Frame(index=index, t=index / meta.fps, image=image)
+            index += 1
+    finally:
+        cap.release()

paperpod/cli.py

@@ -0,0 +1,114 @@
+"""PaperPod command-line interface.
+
+Usage:
+  python -m paperpod probe <video>
+  python -m paperpod detect <video> [--out DIR] [--config config.yaml]
+  python -m paperpod extract-audio <video> [--out FILE] [--config config.yaml]
+"""
+
+from __future__ import annotations
+
+import argparse
+import sys
+from pathlib import Path
+
+from paperpod.capture.audio import extract_audio
+from paperpod.capture.video import probe_video
+from paperpod.config import load_config
+from paperpod.pipeline import run_detection
+
+
+def _add_config_arg(parser: argparse.ArgumentParser) -> None:
+    parser.add_argument(
+        "--config",
+        default=None,
+        help="Path to config.yaml (defaults to ./config.yaml if present)",
+    )
+
+
+def _resolve_config(arg: str | None):
+    if arg is not None:
+        return load_config(arg)
+    default = Path("config.yaml")
+    return load_config(default if default.exists() else None)
+
+
+def cmd_probe(args: argparse.Namespace) -> int:
+    meta = probe_video(args.video)
+    print(f"path:      {meta.path}")
+    print(f"fps:       {meta.fps:.3f}")
+    print(f"frames:    {meta.frame_count}")
+    print(f"duration:  {meta.duration_s:.2f}s")
+    print(f"size:      {meta.width}x{meta.height}")
+    return 0
+
+
+def cmd_detect(args: argparse.Namespace) -> int:
+    cfg = _resolve_config(args.config)
+    out_dir = Path(args.out or Path(cfg.output.dir) / Path(args.video).stem)
+    print(f"Analyzing {args.video} -> {out_dir}")
+
+    report = run_detection(args.video, cfg, out_dir)
+
+    windows = report["stable_windows"]
+    found = [w for w in windows if w["document_found"]]
+    print(f"\nVideo: {report['video_meta']['duration_s']}s, "
+          f"{len(report['motion_events'])} motion events, "
+          f"{len(windows)} stable windows, "
+          f"{len(found)} documents detected\n")
+    header = f"{'win':>3}  {'start':>7}  {'end':>7}  {'sharp':>8}  {'doc':>4}  crop"
+    print(header)
+    print("-" * len(header))
+    for w in windows:
+        print(
+            f"{w['window_id']:>3}  "
+            f"{w['t_start']:>7.2f}  "
+            f"{w['t_end']:>7.2f}  "
+            f"{w.get('sharpness', 0):>8.1f}  "
+            f"{'yes' if w['document_found'] else 'no':>4}  "
+            f"{w.get('crop_path', '-')}"
+        )
+    print(f"\nFull report: {out_dir / 'report.json'}")
+    print(f"Motion scores (for threshold tuning): {out_dir / 'motion_scores.csv'}")
+    return 0
+
+
+def cmd_extract_audio(args: argparse.Namespace) -> int:
+    cfg = _resolve_config(args.config)
+    out = Path(args.out or Path(cfg.output.dir) / (Path(args.video).stem + ".wav"))
+    wav = extract_audio(args.video, out, sample_rate=cfg.audio.sample_rate)
+    print(f"Audio written to {wav}")
+    return 0
+
+
+def main(argv: list[str] | None = None) -> int:
+    parser = argparse.ArgumentParser(
+        prog="paperpod",
+        description="Convert an overhead video of documents into cropped, named PDFs.",
+    )
+    sub = parser.add_subparsers(dest="command", required=True)
+
+    p_probe = sub.add_parser("probe", help="Print video metadata")
+    p_probe.add_argument("video")
+    p_probe.set_defaults(func=cmd_probe)
+
+    p_detect = sub.add_parser(
+        "detect", help="Detect stable windows and document candidates"
+    )
+    p_detect.add_argument("video")
+    p_detect.add_argument("--out", default=None, help="Output directory")
+    _add_config_arg(p_detect)
+    p_detect.set_defaults(func=cmd_detect)
+
+    p_audio = sub.add_parser("extract-audio", help="Extract audio track as WAV")
+    p_audio.add_argument("video")
+    p_audio.add_argument("--out", default=None, help="Output WAV path")
+    _add_config_arg(p_audio)
+    p_audio.set_defaults(func=cmd_extract_audio)
+
+    args = parser.parse_args(argv)
+    return args.func(args)
+
+
+if __name__ == "__main__":
+    sys.exit(main())

paperpod/config.py

@@ -0,0 +1,91 @@
+"""Configuration loading with built-in defaults, overridable via config.yaml."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass, field, fields, is_dataclass
+from pathlib import Path
+from typing import Any
+
+import yaml
+
+
+@dataclass
+class CaptureConfig:
+    sample_fps: float = 8.0
+    processing_width: int = 960
+
+
+@dataclass
+class AudioConfig:
+    sample_rate: int = 16000
+
+
+@dataclass
+class MotionConfig:
+    blur_ksize: int = 21
+    pixel_threshold: int = 12
+    threshold: float = 0.02
+    stable_min_duration_s: float = 1.0
+
+
+@dataclass
+class DocumentConfig:
+    min_area_ratio: float = 0.04
+    canny_low: int = 50
+    canny_high: int = 150
+    enhance: bool = False
+
+
+@dataclass
+class SpeechConfig:
+    window_before_s: float = 3.0
+    window_after_s: float = 5.0
+
+
+@dataclass
+class OutputConfig:
+    dir: str = "./output"
+    consume_dir: str | None = None
+
+
+@dataclass
+class Config:
+    capture: CaptureConfig = field(default_factory=CaptureConfig)
+    audio: AudioConfig = field(default_factory=AudioConfig)
+    motion: MotionConfig = field(default_factory=MotionConfig)
+    document: DocumentConfig = field(default_factory=DocumentConfig)
+    speech: SpeechConfig = field(default_factory=SpeechConfig)
+    output: OutputConfig = field(default_factory=OutputConfig)
+
+
+def _merge_into(instance: Any, data: dict[str, Any]) -> Any:
+    """Overlay a dict of overrides onto a dataclass instance, recursively."""
+    for f in fields(instance):
+        if f.name not in data:
+            continue
+        value = data[f.name]
+        current = getattr(instance, f.name)
+        if is_dataclass(current) and isinstance(value, dict):
+            _merge_into(current, value)
+        else:
+            setattr(instance, f.name, value)
+    return instance
+
+
+def load_config(path: str | Path | None = None) -> Config:
+    """Load config.yaml if present; unknown keys are ignored, missing keys use defaults."""
+    cfg = Config()
+    if path is None:
+        return cfg
+    path = Path(path)
+    if not path.exists():
+        raise FileNotFoundError(f"Config file not found: {path}")
+    data = yaml.safe_load(path.read_text()) or {}
+    return _merge_into(cfg, data)
+
+
+def config_to_dict(cfg: Config) -> dict[str, Any]:
+    """Serialize the effective config (for embedding in run reports)."""
+    from dataclasses import asdict
+
+    return asdict(cfg)

paperpod/events/__init__.py

@@ -0,0 +1,6 @@
+"""Module 3 (not built yet): capture-event state machine.
+
+Will consume the stable windows / motion events from the detection report and
+classify transitions: IDLE, DOCUMENT_PLACED, PAGE_FLIPPED, STACK_CLEARED,
+grouping multi-page events under a shared pod_id.
+"""

paperpod/naming/__init__.py

@@ -0,0 +1,5 @@
+"""Module 6 (not built yet): merge speech/OCR metadata into final filenames.
+
+Pattern: YYYY-MM-DD_source_description.pdf, falling back to
+UNSORTED_<timestamp>.pdf, plus a summary CSV per run.
+"""

paperpod/ocr/__init__.py

@@ -0,0 +1,5 @@
+"""Module 5 (not built yet): OCR fallback via Tesseract (pytesseract).
+
+Will extract vendor name, date, and total from crops that have no matching
+speech segment.
+"""

paperpod/pdf/__init__.py

@@ -0,0 +1,2 @@
+"""Module 7 (not built yet): assemble pod page images into PDFs (img2pdf)
+and stage them for the Paperless-ngx consume directory."""

paperpod/pipeline.py

@@ -0,0 +1,162 @@
+"""Detection pipeline: video in, cropped document candidates + JSON report out.
+
+Two sequential passes over the video (no seeking, works on any codec):
+  Pass 1: sample downscaled frames, score motion, find stable/moving segments.
+  Pass 2: within each stable window, pick the sharpest full-resolution frame,
+          detect the document contour, perspective-correct, save the crop.
+"""
+
+from __future__ import annotations
+
+import csv
+import json
+from pathlib import Path
+from typing import Any
+
+import cv2
+import numpy as np
+
+from paperpod.capture.video import iter_frames, probe_video
+from paperpod.config import Config, config_to_dict
+from paperpod.vision.document import detect_document, warp_document
+from paperpod.vision.enhance import enhance_for_ocr
+from paperpod.vision.motion import Segment, find_segments, motion_score, prepare_gray
+from paperpod.vision.sharpness import sharpness_score
+
+
+def run_detection(video_path: str | Path, cfg: Config, out_dir: str | Path) -> dict[str, Any]:
+    """Run the full detect pipeline and write artifacts into out_dir.
+
+    Artifacts:
+      crops/window_NNN.png        perspective-corrected document candidates
+      frames/window_NNN_full.png  the full best frame per window (for debugging)
+      motion_scores.csv           per-sample motion scores (for threshold tuning)
+      report.json                 stable windows, motion events, detections
+    """
+    video_path = Path(video_path)
+    out_dir = Path(out_dir)
+    crops_dir = out_dir / "crops"
+    frames_dir = out_dir / "frames"
+    crops_dir.mkdir(parents=True, exist_ok=True)
+    frames_dir.mkdir(parents=True, exist_ok=True)
+
+    meta = probe_video(video_path)
+
+    # ---- Pass 1: motion scores on downscaled frames -------------------------
+    timestamps: list[float] = []
+    scores: list[float] = []
+    prev_gray: np.ndarray | None = None
+    small_size: tuple[int, int] | None = None  # (w, h) of processing frames
+
+    for frame in iter_frames(
+        video_path,
+        sample_fps=cfg.capture.sample_fps,
+        resize_width=cfg.capture.processing_width,
+    ):
+        gray = prepare_gray(frame.image, cfg.motion.blur_ksize)
+        if small_size is None:
+            small_size = (frame.image.shape[1], frame.image.shape[0])
+        # First frame has no predecessor; treat it as perfectly stable.
+        score = (
+            0.0
+            if prev_gray is None
+            else motion_score(prev_gray, gray, cfg.motion.pixel_threshold)
+        )
+        timestamps.append(frame.t)
+        scores.append(score)
+        prev_gray = gray
+
+    segments = find_segments(
+        timestamps, scores, cfg.motion.threshold, cfg.motion.stable_min_duration_s
+    )
+    stable_windows = [s for s in segments if s.kind == "stable"]
+
+    with open(out_dir / "motion_scores.csv", "w", newline="") as f:
+        writer = csv.writer(f)
+        writer.writerow(["t", "score"])
+        writer.writerows(zip(timestamps, scores))
+
+    # ---- Pass 2: best frame per stable window, document detection -----------
+    # Map each sampled timestamp range to its stable window for quick lookup.
+    window_records: list[dict[str, Any]] = []
+    best_frames: dict[int, tuple[float, Any]] = {}  # window_idx -> (sharpness, frame)
+
+    def window_for(t: float) -> int | None:
+        for i, w in enumerate(stable_windows):
+            if w.t_start <= t <= w.t_end:
+                return i
+        return None
+
+    if stable_windows:
+        for frame in iter_frames(video_path, sample_fps=cfg.capture.sample_fps):
+            idx = window_for(frame.t)
+            if idx is None:
+                continue
+            score = sharpness_score(frame.image)
+            if idx not in best_frames or score > best_frames[idx][0]:
+                best_frames[idx] = (score, frame)
+
+    for i, window in enumerate(stable_windows):
+        record: dict[str, Any] = {
+            "window_id": i,
+            "t_start": round(window.t_start, 3),
+            "t_end": round(window.t_end, 3),
+            "duration_s": round(window.duration, 3),
+            "document_found": False,
+        }
+        if i in best_frames:
+            sharp, frame = best_frames[i]
+            record["best_frame_t"] = round(frame.t, 3)
+            record["best_frame_index"] = frame.index
+            record["sharpness"] = round(sharp, 1)
+
+            frame_path = frames_dir / f"window_{i:03d}_full.png"
+            cv2.imwrite(str(frame_path), frame.image)
+            record["frame_path"] = str(frame_path.relative_to(out_dir))
+
+            detection = detect_document(
+                frame.image,
+                min_area_ratio=cfg.document.min_area_ratio,
+                canny_low=cfg.document.canny_low,
+                canny_high=cfg.document.canny_high,
+            )
+            if detection is not None:
+                crop = warp_document(frame.image, detection.quad)
+                if cfg.document.enhance:
+                    crop = enhance_for_ocr(crop)
+                crop_path = crops_dir / f"window_{i:03d}.png"
+                cv2.imwrite(str(crop_path), crop)
+                record.update(
+                    document_found=True,
+                    detection_method=detection.method,
+                    area_ratio=round(detection.area_ratio, 4),
+                    quad=[[round(float(x), 1), round(float(y), 1)] for x, y in detection.quad],
+                    crop_path=str(crop_path.relative_to(out_dir)),
+                    crop_size=[crop.shape[1], crop.shape[0]],
+                )
+        window_records.append(record)
+
+    report = {
+        "video": str(video_path),
+        "video_meta": {
+            "fps": round(meta.fps, 3),
+            "frame_count": meta.frame_count,
+            "duration_s": round(meta.duration_s, 3),
+            "width": meta.width,
+            "height": meta.height,
+        },
+        "config": config_to_dict(cfg),
+        "motion_events": [
+            {
+                "kind": s.kind,
+                "t_start": round(s.t_start, 3),
+                "t_end": round(s.t_end, 3),
+                "duration_s": round(s.duration, 3),
+            }
+            for s in segments
+        ],
+        "stable_windows": window_records,
+    }
+    with open(out_dir / "report.json", "w") as f:
+        json.dump(report, f, indent=2)
+    return report

paperpod/review_cli/__init__.py

@@ -0,0 +1,2 @@
+"""Module 8 (not built yet): pre-export review of detected pods
+(rename / merge / split) before anything is written to the consume folder."""

paperpod/transcribe/__init__.py

@@ -0,0 +1,6 @@
+"""Module 4 (not built yet): local speech transcription via faster-whisper.
+
+Will transcribe the WAV produced by capture.extract_audio into timestamped
+segments and match them to capture events within the configured speech window.
+(Named `transcribe` rather than `audio` to avoid clashing with audio capture.)
+"""

paperpod/vision/__init__.py

@@ -0,0 +1,23 @@
+"""Module 2: motion analysis, document detection, perspective correction."""
+
+from paperpod.vision.document import DocumentDetection, detect_document, warp_document
+from paperpod.vision.enhance import enhance_for_ocr
+from paperpod.vision.motion import (
+    Segment,
+    find_segments,
+    motion_score,
+    prepare_gray,
+)
+from paperpod.vision.sharpness import sharpness_score
+
+__all__ = [
+    "Segment",
+    "find_segments",
+    "motion_score",
+    "prepare_gray",
+    "DocumentDetection",
+    "detect_document",
+    "warp_document",
+    "sharpness_score",
+    "enhance_for_ocr",
+]

paperpod/vision/document.py

@@ -0,0 +1,99 @@
+"""Document/receipt contour detection and perspective correction."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import cv2
+import numpy as np
+
+
+@dataclass
+class DocumentDetection:
+    quad: np.ndarray  # 4x2 float32, corner points in source-image coordinates
+    area_ratio: float  # contour area / frame area
+    method: str  # "quad" (clean 4-corner polygon) | "min_rect" (fallback)
+
+
+def _order_points(pts: np.ndarray) -> np.ndarray:
+    """Order 4 points as top-left, top-right, bottom-right, bottom-left."""
+    pts = pts.astype(np.float32)
+    s = pts.sum(axis=1)
+    d = np.diff(pts, axis=1).ravel()
+    return np.array(
+        [
+            pts[np.argmin(s)],  # top-left: smallest x+y
+            pts[np.argmin(d)],  # top-right: smallest y-x
+            pts[np.argmax(s)],  # bottom-right: largest x+y
+            pts[np.argmax(d)],  # bottom-left: largest y-x
+        ],
+        dtype=np.float32,
+    )
+
+
+def detect_document(
+    image_bgr: np.ndarray,
+    min_area_ratio: float = 0.04,
+    canny_low: int = 50,
+    canny_high: int = 150,
+) -> DocumentDetection | None:
+    """Find the largest document-like contour in the frame.
+
+    Pipeline: grayscale -> blur -> Canny -> dilate -> contours -> largest
+    contour above min_area_ratio. If its polygon approximation has 4 corners
+    we use it directly; otherwise we fall back to the minimum-area rectangle.
+    """
+    h, w = image_bgr.shape[:2]
+    frame_area = float(h * w)
+
+    gray = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2GRAY)
+    gray = cv2.GaussianBlur(gray, (5, 5), 0)
+    edges = cv2.Canny(gray, canny_low, canny_high)
+    # Close gaps in the document outline so it forms one contour.
+    edges = cv2.dilate(edges, np.ones((5, 5), np.uint8), iterations=2)
+
+    contours, _ = cv2.findContours(edges, cv2.RETR_EXTERNAL, cv2.CHAIN_APPROX_SIMPLE)
+    if not contours:
+        return None
+
+    best = max(contours, key=cv2.contourArea)
+    area_ratio = cv2.contourArea(best) / frame_area
+    if area_ratio < min_area_ratio:
+        return None
+
+    peri = cv2.arcLength(best, True)
+    approx = cv2.approxPolyDP(best, 0.02 * peri, True)
+    if len(approx) == 4:
+        quad = _order_points(approx.reshape(4, 2))
+        method = "quad"
+    else:
+        rect = cv2.minAreaRect(best)
+        quad = _order_points(cv2.boxPoints(rect))
+        method = "min_rect"
+
+    return DocumentDetection(quad=quad, area_ratio=float(area_ratio), method=method)
+
+
+def warp_document(image_bgr: np.ndarray, quad: np.ndarray) -> np.ndarray:
+    """Perspective-correct the quad region into a clean top-down crop."""
+    quad = _order_points(np.asarray(quad, dtype=np.float32))
+    tl, tr, br, bl = quad
+
+    width = int(max(np.linalg.norm(br - bl), np.linalg.norm(tr - tl)))
+    height = int(max(np.linalg.norm(tr - br), np.linalg.norm(tl - bl)))
+    width, height = max(width, 1), max(height, 1)
+
+    dst = np.array(
+        [[0, 0], [width - 1, 0], [width - 1, height - 1], [0, height - 1]],
+        dtype=np.float32,
+    )
+    matrix = cv2.getPerspectiveTransform(quad, dst)
+    return cv2.warpPerspective(image_bgr, matrix, (width, height))
+
+
+def scale_quad(quad: np.ndarray, scale_x: float, scale_y: float) -> np.ndarray:
+    """Rescale a quad detected on a downscaled frame back to full resolution."""
+    quad = np.asarray(quad, dtype=np.float32).copy()
+    quad[:, 0] *= scale_x
+    quad[:, 1] *= scale_y
+    return quad

paperpod/vision/enhance.py

@@ -0,0 +1,20 @@
+"""Optional image enhancement for OCR-readiness."""
+
+from __future__ import annotations
+
+import cv2
+import numpy as np
+
+
+def enhance_for_ocr(image_bgr: np.ndarray) -> np.ndarray:
+    """Boost local contrast (CLAHE on the luminance channel).
+
+    Deliberately conservative: no binarization, so the crop stays pleasant to
+    read in the PDF while giving OCR more to work with. Thresholding can be
+    added behind a config flag later if OCR needs it.
+    """
+    lab = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2LAB)
+    l_chan, a_chan, b_chan = cv2.split(lab)
+    clahe = cv2.createCLAHE(clipLimit=2.0, tileGridSize=(8, 8))
+    lab = cv2.merge((clahe.apply(l_chan), a_chan, b_chan))
+    return cv2.cvtColor(lab, cv2.COLOR_LAB2BGR)

paperpod/vision/motion.py

@@ -0,0 +1,94 @@
+"""Frame-to-frame motion detection and stable/moving segmentation."""
+
+from __future__ import annotations
+
+from dataclasses import dataclass
+
+import cv2
+import numpy as np
+
+
+def prepare_gray(image_bgr: np.ndarray, blur_ksize: int = 21) -> np.ndarray:
+    """Grayscale + Gaussian blur, the representation used for differencing."""
+    if blur_ksize % 2 == 0:
+        blur_ksize += 1
+    gray = cv2.cvtColor(image_bgr, cv2.COLOR_BGR2GRAY)
+    return cv2.GaussianBlur(gray, (blur_ksize, blur_ksize), 0)
+
+
+def motion_score(
+    prev_gray: np.ndarray, gray: np.ndarray, pixel_threshold: int = 12
+) -> float:
+    """Fraction of pixels (0.0-1.0) whose gray level changed by more than
+    pixel_threshold between consecutive frames.
+
+    A changed-pixel ratio separates localized motion (a hand flipping a page)
+    from sensor noise far better than the mean absolute difference, which
+    dilutes small moving regions across the whole frame.
+    """
+    diff = cv2.absdiff(prev_gray, gray)
+    return float(np.count_nonzero(diff > pixel_threshold)) / diff.size
+
+
+@dataclass
+class Segment:
+    kind: str  # "stable" | "moving"
+    t_start: float
+    t_end: float
+    # Indices into the sampled score series (inclusive range).
+    first_sample: int
+    last_sample: int
+
+    @property
+    def duration(self) -> float:
+        return self.t_end - self.t_start
+
+
+def find_segments(
+    timestamps: list[float],
+    scores: list[float],
+    threshold: float,
+    stable_min_duration_s: float,
+) -> list[Segment]:
+    """Classify each sample as stable/moving and group into segments.
+
+    Stable segments shorter than stable_min_duration_s are reclassified as
+    moving (they are usually mid-gesture pauses), then adjacent segments of
+    the same kind are merged. scores[i] describes the transition into
+    timestamps[i], so both lists must be the same length.
+    """
+    if len(timestamps) != len(scores):
+        raise ValueError("timestamps and scores must have the same length")
+    if not timestamps:
+        return []
+
+    labels = ["stable" if s < threshold else "moving" for s in scores]
+
+    segments: list[Segment] = []
+    start = 0
+    for i in range(1, len(labels) + 1):
+        if i == len(labels) or labels[i] != labels[start]:
+            segments.append(
+                Segment(
+                    kind=labels[start],
+                    t_start=timestamps[start],
+                    t_end=timestamps[i - 1],
+                    first_sample=start,
+                    last_sample=i - 1,
+                )
+            )
+            start = i
+
+    # Demote too-short stable segments, then merge same-kind neighbors.
+    for seg in segments:
+        if seg.kind == "stable" and seg.duration < stable_min_duration_s:
+            seg.kind = "moving"
+
+    merged: list[Segment] = []
+    for seg in segments:
+        if merged and merged[-1].kind == seg.kind:
+            merged[-1].t_end = seg.t_end
+            merged[-1].last_sample = seg.last_sample
+        else:
+            merged.append(seg)
+    return merged

paperpod/vision/sharpness.py

@@ -0,0 +1,13 @@
+"""Blur/sharpness scoring for best-frame selection."""
+
+from __future__ import annotations
+
+import cv2
+import numpy as np
+
+
+def sharpness_score(image: np.ndarray) -> float:
+    """Variance of the Laplacian; higher = sharper."""
+    if image.ndim == 3:
+        image = cv2.cvtColor(image, cv2.COLOR_BGR2GRAY)
+    return float(cv2.Laplacian(image, cv2.CV_64F).var())

requirements.txt

@@ -0,0 +1,17 @@
+# Core (modules 1-2: capture + vision)
+opencv-python>=4.9
+numpy>=1.26
+PyYAML>=6.0
+
+# Module 4: local speech-to-text (installed now so the venv is ready later)
+faster-whisper>=1.0
+
+# Module 5: OCR fallback (requires the tesseract binary: `brew install tesseract`)
+pytesseract>=0.3.10
+
+# Module 7: PDF assembly
+img2pdf>=0.5
+Pillow>=10.0
+
+# Dev / testing
+pytest>=8.0

sample_data/make_sample_video.py

@@ -0,0 +1,128 @@
+"""Generate a synthetic overhead-table test video for pipeline verification.
+
+Timeline (default 24s @ 24fps, 1280x720):
+  0-3s    empty table
+  3-4s    hand slides receipt 1 in (motion)
+  4-9s    receipt 1 stable
+  9-10s   receipt 1 removed (motion)
+  10-12s  empty table
+  12-13s  document 2 placed (motion)
+  13-17s  document 2 page 1 stable
+  17-18s  page flip (content changes, hand passes over)
+  18-22s  document 2 page 2 stable
+  22-23s  document 2 removed
+  23-24s  empty table
+
+Usage: python sample_data/make_sample_video.py [output.mp4]
+"""
+
+from __future__ import annotations
+
+import sys
+from pathlib import Path
+
+import cv2
+import numpy as np
+
+FPS = 24
+W, H = 1280, 720
+RNG = np.random.default_rng(42)
+
+
+def table_background() -> np.ndarray:
+    """Wood-ish brown table with mild noise so it isn't perfectly uniform."""
+    bg = np.full((H, W, 3), (60, 90, 120), dtype=np.uint8)  # BGR brownish
+    noise = RNG.integers(-8, 8, size=(H, W, 1), dtype=np.int16)
+    return np.clip(bg.astype(np.int16) + noise, 0, 255).astype(np.uint8)
+
+
+def make_document(w: int, h: int, lines: int, seed: int) -> np.ndarray:
+    """White page with dark text-like lines."""
+    rng = np.random.default_rng(seed)
+    doc = np.full((h, w, 3), 245, dtype=np.uint8)
+    y = int(h * 0.12)
+    for _ in range(lines):
+        x0 = int(w * 0.1)
+        x1 = int(w * rng.uniform(0.5, 0.9))
+        cv2.line(doc, (x0, y), (x1, y), (30, 30, 30), thickness=max(2, h // 80))
+        y += int(h * 0.08)
+        if y > h * 0.92:
+            break
+    cv2.rectangle(doc, (0, 0), (w - 1, h - 1), (180, 180, 180), 2)
+    return doc
+
+
+def paste_rotated(canvas: np.ndarray, doc: np.ndarray, center: tuple[int, int], angle: float) -> None:
+    """Paste doc onto canvas rotated by angle degrees around its center."""
+    dh, dw = doc.shape[:2]
+    big = np.zeros((H, W, 3), dtype=np.uint8)
+    mask = np.zeros((H, W), dtype=np.uint8)
+    x0, y0 = center[0] - dw // 2, center[1] - dh // 2
+    x0 = max(0, min(W - dw, x0))
+    y0 = max(0, min(H - dh, y0))
+    big[y0 : y0 + dh, x0 : x0 + dw] = doc
+    mask[y0 : y0 + dh, x0 : x0 + dw] = 255
+    matrix = cv2.getRotationMatrix2D((x0 + dw / 2, y0 + dh / 2), angle, 1.0)
+    big = cv2.warpAffine(big, matrix, (W, H))
+    mask = cv2.warpAffine(mask, matrix, (W, H))
+    canvas[mask > 0] = big[mask > 0]
+
+
+def draw_hand(canvas: np.ndarray, t01: float, entering: bool) -> None:
+    """A skin-toned blob sweeping across the frame to create motion."""
+    progress = t01 if entering else 1.0 - t01
+    cx = int(W * (1.1 - progress * 0.6))
+    cy = int(H * 0.5 + 60 * np.sin(progress * np.pi))
+    cv2.ellipse(canvas, (cx, cy), (110, 70), 20, 0, 360, (120, 150, 200), -1)
+
+
+def render(out_path: Path) -> None:
+    receipt = make_document(280, 520, lines=10, seed=1)
+    page1 = make_document(560, 420, lines=8, seed=2)
+    page2 = make_document(560, 420, lines=8, seed=3)
+    bg = table_background()
+
+    writer = cv2.VideoWriter(
+        str(out_path), cv2.VideoWriter_fourcc(*"mp4v"), FPS, (W, H)
+    )
+    duration = 24
+    for i in range(duration * FPS):
+        t = i / FPS
+        frame = bg.copy()
+
+        if 3 <= t < 4:  # receipt entering
+            draw_hand(frame, t - 3, entering=True)
+            paste_rotated(frame, receipt, (int(W * (1.2 - (t - 3) * 0.7)), 360), 4)
+        elif 4 <= t < 9:  # receipt stable
+            paste_rotated(frame, receipt, (int(W * 0.5), 360), 4)
+        elif 9 <= t < 10:  # receipt leaving
+            paste_rotated(frame, receipt, (int(W * (0.5 - (t - 9) * 0.7)), 360), 4)
+            draw_hand(frame, t - 9, entering=False)
+        elif 12 <= t < 13:  # doc2 entering
+            draw_hand(frame, t - 12, entering=True)
+            paste_rotated(frame, page1, (int(W * (1.2 - (t - 12) * 0.7)), 360), -3)
+        elif 13 <= t < 17:  # doc2 page 1 stable
+            paste_rotated(frame, page1, (int(W * 0.5), 360), -3)
+        elif 17 <= t < 18:  # page flip
+            paste_rotated(frame, page1 if t < 17.5 else page2, (int(W * 0.5), 360), -3)
+            draw_hand(frame, (t - 17), entering=t < 17.5)
+        elif 18 <= t < 22:  # doc2 page 2 stable
+            paste_rotated(frame, page2, (int(W * 0.5), 360), -3)
+        elif 22 <= t < 23:  # doc2 leaving
+            paste_rotated(frame, page2, (int(W * (0.5 - (t - 22) * 0.7)), 360), -3)
+            draw_hand(frame, t - 22, entering=False)
+        # else: empty table
+
+        # Slight per-frame sensor noise so "stable" is realistic, not identical.
+        noise = RNG.integers(-2, 2, size=(H, W, 1), dtype=np.int16)
+        frame = np.clip(frame.astype(np.int16) + noise, 0, 255).astype(np.uint8)
+        writer.write(frame)
+    writer.release()
+    print(f"Wrote {out_path} ({duration}s @ {FPS}fps, {W}x{H})")
+
+
+if __name__ == "__main__":
+    default = Path(__file__).parent / "videos" / "synthetic_sample.mp4"
+    out = Path(sys.argv[1]) if len(sys.argv) > 1 else default
+    out.parent.mkdir(parents=True, exist_ok=True)
+    render(out)

tests/test_capture.py

@@ -0,0 +1,47 @@
+import cv2
+import numpy as np
+import pytest
+
+from paperpod.capture.video import iter_frames, probe_video
+
+
+@pytest.fixture(scope="module")
+def tiny_video(tmp_path_factory):
+    """3s, 24fps, 320x240 video with a moving square."""
+    path = tmp_path_factory.mktemp("video") / "tiny.mp4"
+    writer = cv2.VideoWriter(str(path), cv2.VideoWriter_fourcc(*"mp4v"), 24, (320, 240))
+    for i in range(72):
+        frame = np.full((240, 320, 3), 100, dtype=np.uint8)
+        x = 10 + i * 3
+        cv2.rectangle(frame, (x, 100), (x + 40, 140), (255, 255, 255), -1)
+        writer.write(frame)
+    writer.release()
+    return path
+
+
+def test_probe(tiny_video):
+    meta = probe_video(tiny_video)
+    assert meta.fps == pytest.approx(24, abs=0.5)
+    assert meta.frame_count == 72
+    assert meta.width == 320 and meta.height == 240
+    assert meta.duration_s == pytest.approx(3.0, abs=0.1)
+
+
+def test_iter_frames_sample_rate(tiny_video):
+    frames = list(iter_frames(tiny_video, sample_fps=8))
+    # 24fps sampled at 8fps -> every 3rd frame -> 24 frames
+    assert len(frames) == 24
+    ts = [f.t for f in frames]
+    assert ts == sorted(ts)
+    assert ts[0] == 0.0
+
+
+def test_iter_frames_resize(tiny_video):
+    frame = next(iter_frames(tiny_video, sample_fps=8, resize_width=160))
+    assert frame.image.shape[1] == 160
+    assert frame.image.shape[0] == 120
+
+
+def test_probe_missing_file():
+    with pytest.raises(FileNotFoundError):
+        probe_video("does_not_exist.mp4")

tests/test_config.py

@@ -0,0 +1,26 @@
+from paperpod.config import load_config
+
+
+def test_defaults_without_file():
+    cfg = load_config(None)
+    assert cfg.capture.sample_fps == 8.0
+    assert cfg.motion.threshold == 0.02
+    assert cfg.motion.pixel_threshold == 12
+
+
+def test_partial_override(tmp_path):
+    p = tmp_path / "config.yaml"
+    p.write_text("motion:\n  threshold: 7.5\n")
+    cfg = load_config(p)
+    assert cfg.motion.threshold == 7.5
+    # untouched keys keep defaults
+    assert cfg.motion.stable_min_duration_s == 1.0
+    assert cfg.capture.sample_fps == 8.0
+
+
+def test_repo_config_loads():
+    from pathlib import Path
+
+    repo_cfg = Path(__file__).parent.parent / "config.yaml"
+    cfg = load_config(repo_cfg)
+    assert cfg.output.dir == "./output"

tests/test_document.py

@@ -0,0 +1,49 @@
+import cv2
+import numpy as np
+
+from paperpod.vision.document import detect_document, warp_document
+from paperpod.vision.sharpness import sharpness_score
+
+
+def synthetic_frame(angle: float = 8.0) -> np.ndarray:
+    """Dark table with a rotated white 400x600 'document'."""
+    frame = np.full((720, 1280, 3), (60, 90, 120), dtype=np.uint8)
+    doc = np.full((600, 400, 3), 245, dtype=np.uint8)
+    canvas = np.zeros_like(frame)
+    mask = np.zeros(frame.shape[:2], dtype=np.uint8)
+    x0, y0 = 440, 60
+    canvas[y0 : y0 + 600, x0 : x0 + 400] = doc
+    mask[y0 : y0 + 600, x0 : x0 + 400] = 255
+    matrix = cv2.getRotationMatrix2D((x0 + 200, y0 + 300), angle, 1.0)
+    canvas = cv2.warpAffine(canvas, matrix, (1280, 720))
+    mask = cv2.warpAffine(mask, matrix, (1280, 720))
+    frame[mask > 0] = canvas[mask > 0]
+    return frame
+
+
+def test_detects_rotated_document():
+    det = detect_document(synthetic_frame())
+    assert det is not None
+    assert det.quad.shape == (4, 2)
+    assert det.area_ratio > 0.2  # 400x600 doc in a 1280x720 frame
+
+
+def test_warp_restores_aspect_ratio():
+    det = detect_document(synthetic_frame(angle=8.0))
+    crop = warp_document(synthetic_frame(angle=8.0), det.quad)
+    h, w = crop.shape[:2]
+    # Source doc is 400x600 (aspect 1.5); allow tolerance for edges/dilation.
+    assert 1.3 < h / w < 1.7
+    # Crop should be mostly white paper.
+    assert crop.mean() > 180
+
+
+def test_no_document_on_empty_table():
+    frame = np.full((720, 1280, 3), (60, 90, 120), dtype=np.uint8)
+    assert detect_document(frame) is None
+
+
+def test_sharpness_prefers_sharp_frame():
+    sharp = synthetic_frame()
+    blurred = cv2.GaussianBlur(sharp, (31, 31), 0)
+    assert sharpness_score(sharp) > sharpness_score(blurred)

tests/test_motion.py

@@ -0,0 +1,44 @@
+import numpy as np
+
+from paperpod.vision.motion import find_segments, motion_score, prepare_gray
+
+
+def series(pattern: list[tuple[float, int]], dt: float = 0.125):
+    """Expand [(score, count), ...] into (timestamps, scores)."""
+    scores = [s for s, n in pattern for _ in range(n)]
+    timestamps = [i * dt for i in range(len(scores))]
+    return timestamps, scores
+
+
+def test_basic_stable_and_moving_segments():
+    # stable 2s, moving 1s, stable 2s
+    ts, sc = series([(0.5, 16), (10.0, 8), (0.5, 16)])
+    segs = find_segments(ts, sc, threshold=4.0, stable_min_duration_s=1.0)
+    kinds = [s.kind for s in segs]
+    assert kinds == ["stable", "moving", "stable"]
+    assert segs[0].duration >= 1.0
+    assert segs[2].duration >= 1.0
+
+
+def test_short_stable_blip_is_demoted_to_moving():
+    # moving, tiny stable pause (0.25s), moving -> should merge into one moving segment
+    ts, sc = series([(10.0, 8), (0.5, 2), (10.0, 8)])
+    segs = find_segments(ts, sc, threshold=4.0, stable_min_duration_s=1.0)
+    assert [s.kind for s in segs] == ["moving"]
+
+
+def test_empty_input():
+    assert find_segments([], [], threshold=4.0, stable_min_duration_s=1.0) == []
+
+
+def test_motion_score_zero_for_identical_frames():
+    img = np.random.default_rng(0).integers(0, 255, (100, 100, 3), dtype=np.uint8)
+    g = prepare_gray(img)
+    assert motion_score(g, g) == 0.0
+
+
+def test_motion_score_positive_for_changed_frames():
+    rng = np.random.default_rng(0)
+    a = prepare_gray(rng.integers(0, 255, (100, 100, 3), dtype=np.uint8))
+    b = prepare_gray(np.full((100, 100, 3), 200, dtype=np.uint8))
+    assert motion_score(a, b) > 0