Author SHA1 Message Date
ilia 332e38dbe8 Merge pull request 'Add AGENTS.md orientation for Cursor agents' (#9) from chore/bootstrap-agents-md into main
CI / skip-ci-check (push) Successful in 34s
CI / secret-scan (push) Successful in 30s
CI / python-ci (push) Successful in 1m42s
2026-08-05 17:51:42 -05:00
ilia 68a14d5d19 Add AGENTS.md orientation for Cursor agents
CI / skip-ci-check (pull_request) Successful in 34s
CI / secret-scan (pull_request) Successful in 29s
CI / python-ci (pull_request) Successful in 2m57s
2026-08-05 18:14:15 -04:00
ilia c8a10b4270 Merge pull request 'Humanize README and docs tone' (#8) from docs/humanize-prose into main
CI / skip-ci-check (push) Successful in 32s
CI / secret-scan (push) Successful in 30s
CI / python-ci (push) Successful in 2m24s
2026-08-05 14:18:01 -05:00
ilia 662cb1b1b8 Humanize READMEs and guides for clearer, professional tone
CI / skip-ci-check (pull_request) Successful in 32s
CI / secret-scan (pull_request) Successful in 30s
CI / python-ci (pull_request) Successful in 2m5s
Align with project-template docs/writing-docs.md: plain openings, no
emoji decoration, archive scratch plans where they cluttered live docs.
2026-08-05 14:52:53 -04:00
ilia d790dc753b Merge pull request 'Add MIT LICENSE' (#7) from chore/add-license into main
CI / skip-ci-check (push) Successful in 19s
CI / secret-scan (push) Successful in 17s
CI / python-ci (push) Successful in 1m10s
2026-07-26 15:43:38 -05:00
ilia 821d4d58d8 Add MIT LICENSE
CI / skip-ci-check (pull_request) Successful in 18s
CI / secret-scan (pull_request) Successful in 17s
CI / python-ci (pull_request) Successful in 1m10s
2026-07-26 15:40:19 -05:00
ilia 1665885927 Merge pull request 'Fix lint debt and make CI gates honest' (#6) from chore/lint-debt-and-honest-ci into main
CI / skip-ci-check (push) Successful in 22s
CI / python-ci (push) Successful in 1m40s
CI / secret-scan (push) Successful in 19s
2026-07-26 14:47:32 -05:00
ilia 5df21d82f4 Fix lint debt and make CI gates honest
CI / skip-ci-check (pull_request) Successful in 19s
CI / secret-scan (pull_request) Successful in 19s
CI / python-ci (pull_request) Successful in 2m38s
- ruff: fix all 328 errors (autofix + manual); move config to [tool.ruff.lint]
- mypy: fix all errors (annotations, nullable-column guards, stale kwargs
  in weekly report caller)
- black: reformat src/tests so black --check passes
- CI: remove `|| true` from ruff/black/mypy/pytest steps in both
  .gitea and .github workflows; install project deps and add mypy
  step in gitea python-ci so gates actually run
- docs: move 14 root status/guide markdown files into docs/, update links
2026-07-26 15:41:07 -04:00
ilia 6e5e69ed10 ci: add local pre-commit gitleaks hook
CI / skip-ci-check (push) Successful in 11s
CI / secret-scan (push) Successful in 11s
CI / python-ci (push) Successful in 42s
2026-07-13 15:00:18 -05:00
ilia 781a45f1a3 Mark deploy and daily email verified (#4)
CI / skip-ci-check (push) Successful in 9s
CI / secret-scan (push) Successful in 8s
CI / python-ci (push) Successful in 22s
2026-07-12 10:38:51 -05:00
ilia 5d1fc601ea Refresh handoff and harden daily report (#3)
CI / skip-ci-check (push) Successful in 17s
CI / secret-scan (push) Successful in 8s
CI / python-ci (push) Successful in 27s
2026-07-12 10:26:17 -05:00
ilia 5c0385e27c Merge pull request 'Add homelab Gitea Actions CI (python-scraper)' (#2) from ci/bootstrap-gitea-actions-python into main
CI / skip-ci-check (push) Successful in 10s
CI / secret-scan (push) Successful in 8s
CI / python-ci (push) Successful in 22s
2026-05-29 21:47:12 -05:00
ilia 8110c5949d ci: sync gitleaks allowlist
CI / skip-ci-check (pull_request) Successful in 10s
CI / secret-scan (pull_request) Successful in 9s
CI / python-ci (pull_request) Successful in 31s
2026-05-29 21:31:14 -05:00
ilia fc510f2b2c ci: refresh workflow (re-run pipelines) 2026-05-29 21:31:13 -05:00
ilia 9cb05ddf77 ci: sync gitleaks allowlist
CI / skip-ci-check (pull_request) Successful in 11s
CI / secret-scan (pull_request) Successful in 10s
CI / python-ci (pull_request) Successful in 35s
2026-05-29 21:27:44 -05:00
ilia 1f4e9c075a ci: refresh workflow (re-run pipelines)
CI / skip-ci-check (pull_request) Successful in 11s
CI / secret-scan (pull_request) Successful in 10s
CI / python-ci (pull_request) Successful in 34s
2026-05-29 21:27:42 -05:00
ilia a95429509f ci: sync gitleaks allowlist
CI / skip-ci-check (pull_request) Successful in 11s
CI / secret-scan (pull_request) Successful in 12s
CI / python-ci (pull_request) Successful in 29s
2026-05-29 21:23:20 -05:00
ilia 169f28363b ci: refresh workflow (re-run pipelines) 2026-05-29 21:23:18 -05:00
ilia b383f9dd8d ci: add homelab gitleaks allowlist
CI / skip-ci-check (pull_request) Successful in 13s
CI / secret-scan (pull_request) Successful in 9s
CI / python-ci (pull_request) Successful in 31s
2026-05-29 21:18:58 -05:00
ilia a8757fd6f1 ci: refresh workflow (re-run pipelines) 2026-05-29 21:18:54 -05:00
ilia 3950867dae ci: refresh workflow (re-run pipelines)
CI / skip-ci-check (pull_request) Successful in 10s
CI / secret-scan (pull_request) Failing after 9s
CI / python-ci (pull_request) Successful in 34s
2026-05-29 20:29:44 -05:00
ilia 1bce7581e5 ci: refresh workflow (re-run pipelines)
CI / skip-ci-check (pull_request) Successful in 10s
CI / python-ci (pull_request) Failing after 10s
CI / secret-scan (pull_request) Failing after 9s
2026-05-29 20:27:19 -05:00
ilia b9a2e1011f ci: sync workflow template
CI / skip-ci-check (pull_request) Successful in 10s
CI / python-ci (pull_request) Failing after 9s
CI / secret-scan (pull_request) Failing after 10s
2026-05-29 20:25:06 -05:00
ilia a11108838d ci: refresh workflow (re-run pipelines)
CI / skip-ci-check (pull_request) Successful in 9s
CI / python-ci (pull_request) Failing after 9s
CI / secret-scan (pull_request) Failing after 9s
2026-05-29 20:19:46 -05:00
ilia 2ee601c198 ci: sync workflow template (node container + host fixes)
CI / skip-ci-check (pull_request) Successful in 10s
CI / python-ci (pull_request) Failing after 14s
CI / secret-scan (pull_request) Failing after 10s
2026-05-29 20:14:05 -05:00
ilia 367d76eb9d ci: add homelab Gitea Actions workflow (ci-python.yml)
CI / skip-ci-check (pull_request) Failing after 8s
CI / python-ci (pull_request) Has been skipped
CI / secret-scan (pull_request) Has been skipped
2026-05-29 16:14:56 -05:00
62 changed files with 1536 additions and 1330 deletions
+84
View File
@@ -0,0 +1,84 @@
---
# ci-sync: 2026-05-30T02:31:20Z
# Homelab CI — Python lane (git-ci-01) + secret scan (git-ci-02)
# Skip: @skipci in branch name or commit message
name: CI
on:
push:
branches: [master, main]
pull_request:
types: [opened, synchronize, reopened]
jobs:
skip-ci-check:
runs-on: [homelab, self-hosted, linux]
container:
image: node:20-bookworm
outputs:
should-skip: ${{ steps.check.outputs.skip }}
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 1
- id: check
run: |
SKIP=0
BRANCH="${GITHUB_HEAD_REF:-${GITHUB_REF#refs/heads/}}"
MSG="${GITHUB_EVENT_HEAD_COMMIT_MESSAGE:-$(git log -1 --pretty=%B 2>/dev/null || true)}"
echo "$BRANCH" "$MSG" | grep -qi '@skipci' && SKIP=1
echo "skip=$SKIP" >> $GITHUB_OUTPUT
python-ci:
needs: skip-ci-check
if: needs.skip-ci-check.outputs.should-skip != '1'
runs-on: [homelab, self-hosted, linux, python]
container:
# node image: actions/checkout@v4 needs Node; install python3 in-job
image: node:20-bookworm
steps:
- uses: actions/checkout@v4
- name: Install Python tooling
run: |
apt-get update -qq
DEBIAN_FRONTEND=noninteractive apt-get install -y -qq python3 python3-pip python3-venv
python3 -m pip install --upgrade pip --break-system-packages
if [ -f requirements.txt ]; then pip install -r requirements.txt --break-system-packages; fi
if [ -f requirements-dev.txt ]; then pip install -r requirements-dev.txt --break-system-packages; fi
# Install the project + dev tools so lint/type/test gates run for real
pip install -e ".[dev]" --break-system-packages
pip install bandit pip-audit --break-system-packages
# Lint, type-check, and tests are hard gates — no `|| true`.
- name: Ruff lint
run: ruff check src tests
- name: Mypy
run: mypy src
- name: Bandit (advisory)
run: bandit -r . -q || true
- name: pip-audit (advisory)
run: pip-audit -r requirements.txt 2>/dev/null || pip-audit 2>/dev/null || true
- name: Pytest
run: pytest -q
secret-scan:
needs: skip-ci-check
if: needs.skip-ci-check.outputs.should-skip != '1'
runs-on: [homelab, self-hosted, linux, heavy]
steps:
- uses: actions/checkout@v4
with:
fetch-depth: 0
- name: Gitleaks
run: |
extra=""
if [ -f .gitleaks.toml ]; then
extra="--config /repo/.gitleaks.toml"
fi
docker run --rm -v "$PWD:/repo" ghcr.io/gitleaks/gitleaks:latest \
detect --source /repo --no-banner --redact ${extra}
+4 -3
View File
@@ -34,14 +34,15 @@ jobs:
.venv/bin/pip install --upgrade pip .venv/bin/pip install --upgrade pip
.venv/bin/pip install -e ".[dev]" .venv/bin/pip install -e ".[dev]"
# Linters are hard gates — no `|| true`.
- name: Run linters - name: Run linters
run: | run: |
echo "Running ruff..." echo "Running ruff..."
.venv/bin/ruff check src/ tests/ || true .venv/bin/ruff check src/ tests/
echo "Running black check..." echo "Running black check..."
.venv/bin/black --check src/ tests/ || true .venv/bin/black --check src/ tests/
echo "Running mypy..." echo "Running mypy..."
.venv/bin/mypy src/ --install-types --non-interactive || true .venv/bin/mypy src/ --install-types --non-interactive
- name: Run tests with coverage - name: Run tests with coverage
env: env:
+30
View File
@@ -0,0 +1,30 @@
# Homelab bootstrap — gitleaks allowlist (tests, examples, placeholders)
#
# IMPORTANT: `useDefault = true` is required — without it gitleaks loads ONLY
# this file (title + allowlist) with ZERO detection rules, so it would never
# flag a real secret. Fixed 2026-07 (security-hardening track); if you're
# re-pushing this template to a repo that already had the old version, that
# repo's secret scanning was a no-op until this lands.
title = "homelab gitea bootstrap"
[extend]
useDefault = true
[allowlist]
description = "Test fixtures and example configs are not production secrets"
paths = [
'''(?i).*\.test\.(ts|tsx|js|jsx|py)$''',
'''(?i).*\.spec\.(ts|tsx|js|jsx)$''',
'''(?i).*/tests/.*''',
'''(?i).*/__tests__/.*''',
'''(?i).*\.example\.(yml|yaml|env|json|toml)$''',
'''(?i).*vault\.example\.(yml|yaml)$''',
'''(?i).*\.env\.example$''',
'''(?i)CUSTOMIZATION_CHECKLIST\.md$''',
]
regexes = [
'''(?i)(invalid|fake|dummy|placeholder|example|changeme|change_me|not-a-real)''',
'''(?i)sk-or-invalid''',
'''(?i)msk-or-invalid''',
'''(?i)your_ssh_private_key_here''',
]
+20
View File
@@ -0,0 +1,20 @@
# AGENTS.md — POTE
Short orientation for Cursor agents. Prefer this over rediscovering the repo.
## Defaults
- Read `README.md` for run/install; use `make help` when a Makefile exists.
- Lint and tests are **hard gates** in CI — never add `|| true` to them.
- **Do not** commit secrets; app creds live in Infisical (`secrets.levkin.ca`, `/apps/POTE`).
- Multi-step or ambiguous work → Plan mode first, then Agent mode.
## Docs voice
READMEs/guides: `~/Documents/code/project-template/docs/writing-docs.md` (no emoji decoration).
## Close-out
1. Run the narrowest verify that fits (`make test`, `npm test`, `pytest`, CI workflow locally).
2. Public GitHub mirror (`Gitilia/*`): after README/docs release merges, sync mirrors — see `ansible/docs/guides/github-mirrors.md`.
3. Non-trivial PR: offer review; merge only when the user asks.
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Ilia Dobkin
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+37 -172
View File
@@ -1,202 +1,67 @@
# POTE Public Officials Trading Explorer # POTE (Public Officials Trading Explorer)
**Research-only tool for tracking and analyzing public stock trades by government officials.** Research tool for tracking publicly disclosed stock trades by government
officials (starting with U.S. Congress). Computes descriptive metrics and
risk/ethics flags from lawfully available public data.
⚠️ **Important**: This project is for personal research and transparency analysis only. It is **NOT** for investment advice or live trading. Not investment advice. Not for live trading. Public disclosures only; data
may be delayed or incomplete. No claims about inside information.
## What is this? Status: active. Homelab LXC deploy documented under `docs/`.
POTE tracks stock trading activity of government officials (starting with U.S. Congress) using lawfully available public data sources. It computes research metrics, descriptive signals, and risk/ethics flags to help understand trading patterns.
## Key constraints
- **Public data only**: House Stock Watcher (free!), yfinance (free!), QuiverQuant/FMP (optional)
- **Research framing**: All outputs are descriptive analytics, not trading recommendations
- **No inside information claims**: We use public disclosures that may be delayed or incomplete
## Current Status
**PR1 Complete**: Project scaffold, DB models, price loader
**PR2 Complete**: Congressional trade ingestion (House Stock Watcher)
**PR3 Complete**: Security enrichment + deployment infrastructure
**PR4 Complete**: Phase 2 analytics - returns, benchmarks, performance metrics
**45+ passing tests, 88%+ coverage**
## Quick start ## Quick start
**🚀 Already deployed?** See **[QUICKSTART.md](QUICKSTART.md)** for full usage guide!
**📦 Deploying?** See **[PROXMOX_QUICKSTART.md](PROXMOX_QUICKSTART.md)** for Proxmox LXC deployment (recommended).
**📧 Want automated reports?** See **[AUTOMATION_QUICKSTART.md](AUTOMATION_QUICKSTART.md)** for email reporting setup!
### Local Development
```bash ```bash
# Install git clone <repository-url>
git clone <your-repo> cd pote # or POTE
cd pote
make install make install
source venv/bin/activate source venv/bin/activate
# Run migrations
make migrate make migrate
python scripts/ingest_from_fixtures.py # offline sample
make test
```
# Ingest sample data (offline, for testing) With network:
python scripts/ingest_from_fixtures.py
# Enrich securities with company info ```bash
python scripts/enrich_securities.py
# With internet:
python scripts/fetch_congressional_trades.py python scripts/fetch_congressional_trades.py
python scripts/fetch_sample_prices.py python scripts/fetch_sample_prices.py
# Run tests
make test
# Lint & format
make lint format
``` ```
### Production Deployment Deploy: `bash scripts/proxmox_setup.sh` or `docker-compose up -d`.
Ops handoff: [docs/HANDOFF-2026-05-27.md](docs/HANDOFF-2026-05-27.md).
## Useful commands
```bash ```bash
# Proxmox LXC (Recommended - 5 minutes)
bash scripts/proxmox_setup.sh
# Docker
docker-compose up -d
```
## Tech stack
- **Language**: Python 3.10+
- **Database**: PostgreSQL or SQLite (dev)
- **Data**: House Stock Watcher (free!), yfinance (free!), QuiverQuant/FMP (optional)
- **Libraries**: SQLAlchemy, Alembic, pandas, numpy, httpx, yfinance, scikit-learn
- **Testing**: pytest (28 tests, 87%+ coverage)
## Documentation
**Getting Started**:
- [`README.md`](README.md) This file
- [`QUICKSTART.md`](QUICKSTART.md) **How to use your deployed POTE instance**
- [`STATUS.md`](STATUS.md) Current project status
- [`FREE_TESTING_QUICKSTART.md`](FREE_TESTING_QUICKSTART.md) Test for $0
- [`OFFLINE_DEMO.md`](OFFLINE_DEMO.md) Works without internet!
**Deployment**:
- [`PROXMOX_QUICKSTART.md`](PROXMOX_QUICKSTART.md) **Proxmox quick deployment (5 min)**
- [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md) **Automated reporting setup (5 min)**
- [`docs/07_deployment.md`](docs/07_deployment.md) Full deployment guide (all platforms)
- [`docs/08_proxmox_deployment.md`](docs/08_proxmox_deployment.md) Proxmox detailed guide
- [`docs/12_automation_and_reporting.md`](docs/12_automation_and_reporting.md) Automation & CI/CD guide
- [`Dockerfile`](Dockerfile) + [`docker-compose.yml`](docker-compose.yml) Docker setup
**Technical**:
- [`docs/00_mvp.md`](docs/00_mvp.md) MVP roadmap
- [`docs/01_architecture.md`](docs/01_architecture.md) Architecture
- [`docs/02_data_model.md`](docs/02_data_model.md) Database schema
- [`docs/03_data_sources.md`](docs/03_data_sources.md) Data sources
- [`docs/04_safety_ethics.md`](docs/04_safety_ethics.md) Research-only guardrails
- [`docs/05_dev_setup.md`](docs/05_dev_setup.md) Dev conventions
- [`docs/06_free_testing_data.md`](docs/06_free_testing_data.md) Testing strategies
**PR Summaries**:
- [`docs/PR1_SUMMARY.md`](docs/PR1_SUMMARY.md) Scaffold + price loader
- [`docs/PR2_SUMMARY.md`](docs/PR2_SUMMARY.md) Congressional trades
- [`docs/PR3_SUMMARY.md`](docs/PR3_SUMMARY.md) Enrichment + deployment
- [`docs/PR4_SUMMARY.md`](docs/PR4_SUMMARY.md) **Analytics foundation (returns, benchmarks, metrics)**
## What's Working Now
- ✅ SQLAlchemy models for officials, securities, trades, prices
- ✅ Alembic migrations
- ✅ Price loader with yfinance (idempotent, upsert)
- ✅ Congressional trade ingestion from House Stock Watcher (FREE!)
- ✅ Security enrichment (company names, sectors, industries)
- ✅ ETL to populate officials & trades tables
- ✅ Docker + deployment infrastructure
- ✅ 93 passing tests with 88%+ coverage
- ✅ Linting (ruff + mypy) all green
- ✅ Works 100% offline with fixtures
- ✅ Real-time market monitoring & alert system
- ✅ Disclosure timing correlation engine
- ✅ Pattern detection & comparative analysis
- ✅ Automated email reporting (daily/weekly)
- ✅ CI/CD pipeline (GitHub/Gitea Actions)
## What You Can Do Now
### Analyze Performance
```bash
# Analyze specific official
python scripts/analyze_official.py "Nancy Pelosi" --window 90 python scripts/analyze_official.py "Nancy Pelosi" --window 90
# System-wide analysis
python scripts/calculate_all_returns.py python scripts/calculate_all_returns.py
```
### Market Monitoring
```bash
# Run market scan
python scripts/monitor_market.py --scan python scripts/monitor_market.py --scan
# Analyze timing of recent disclosures
python scripts/analyze_disclosure_timing.py --recent 7
# Generate pattern report
python scripts/generate_pattern_report.py --days 365
```
### Automated Reporting
```bash
# Set up daily/weekly email reports (5 minutes!)
./scripts/setup_cron.sh
# Send manual report
python scripts/send_daily_report.py --to your@email.com python scripts/send_daily_report.py --to your@email.com
``` ```
### Add More Data ## Stack
```bash
# Manual entry
python scripts/add_custom_trades.py
# CSV import Python 3.10+, SQLAlchemy, Alembic, pandas/numpy, httpx, yfinance.
python scripts/scrape_alternative_sources.py import trades.csv PostgreSQL or SQLite for local. pytest + ruff + mypy.
```
## System Architecture Data sources: House Stock Watcher, yfinance; QuiverQuant/FMP optional.
POTE now includes a complete 3-phase monitoring system: ## Docs
**Phase 1: Real-Time Market Monitoring** | Doc | Purpose |
- Tracks ~50 most-traded congressional stocks |-----|---------|
- Detects unusual volume, price spikes, volatility | [docs/QUICKSTART.md](docs/QUICKSTART.md) | Using a deployed instance |
- Logs all alerts with timestamps and severity | [docs/PROXMOX_QUICKSTART.md](docs/PROXMOX_QUICKSTART.md) | Proxmox LXC |
| [docs/AUTOMATION_QUICKSTART.md](docs/AUTOMATION_QUICKSTART.md) | Email reports |
| [docs/01_architecture.md](docs/01_architecture.md) | Architecture |
| [docs/02_data_model.md](docs/02_data_model.md) | Schema |
| [docs/04_safety_ethics.md](docs/04_safety_ethics.md) | Research guardrails |
| [docs/07_deployment.md](docs/07_deployment.md) | Full deploy |
| [docs/00_mvp.md](docs/00_mvp.md) | Roadmap |
**Phase 2: Disclosure Correlation** PR writeups live under `docs/archive/` when moved.
- Matches trades with prior market alerts (30-45 day lookback)
- Calculates "timing advantage score" (0-100)
- Identifies suspicious timing patterns
**Phase 3: Pattern Detection** ## License
- Ranks officials by consistent suspicious timing
- Analyzes by ticker, sector, and political party
- Generates comprehensive reports
**Full Documentation**: See [`MONITORING_SYSTEM_COMPLETE.md`](MONITORING_SYSTEM_COMPLETE.md) MIT for research/educational use. Not investment advice.
## Next Steps
- [ ] Signals: "follow_research", "avoid_risk", "watch" with confidence scores
- [ ] Clustering: group officials by trading behavior patterns
- [ ] API: FastAPI backend for queries
- [ ] Dashboard: React/Streamlit visualization
See [`docs/00_mvp.md`](docs/00_mvp.md) for the full roadmap.
---
**License**: MIT (for research/educational use only)
**Disclaimer**: Not investment advice. Use public data only. No claims about inside information.
+4 -4
View File
@@ -6,7 +6,7 @@ Great question! Here are multiple strategies for testing the full pipeline **wit
--- ---
## Strategy 1: Mock/Fixture Data (Current Approach ) ## Strategy 1: Mock/Fixture Data (Current Approach )
**What we already have:** **What we already have:**
- `tests/conftest.py` creates in-memory SQLite DB with sample officials, securities, trades - `tests/conftest.py` creates in-memory SQLite DB with sample officials, securities, trades
@@ -99,7 +99,7 @@ def test_etl_with_real_data():
--- ---
## Strategy 4: Hybrid Testing (Recommended 🌟) ## Strategy 4: Hybrid Testing (Recommended )
**Combine all strategies**: **Combine all strategies**:
@@ -164,7 +164,7 @@ def test_etl_with_real_data():
| Source | Free Tier | Paid Tier | Best For | | Source | Free Tier | Paid Tier | Best For |
|--------|-----------|-----------|----------| |--------|-----------|-----------|----------|
| **yfinance** | Unlimited | N/A | Prices (already working ) | | **yfinance** | Unlimited | N/A | Prices (already working ) |
| **House Stock Watcher** | Unlimited scraping | N/A | Free trades (best option) | | **House Stock Watcher** | Unlimited scraping | N/A | Free trades (best option) |
| **Quiver Free** | 500 calls/mo | $30/mo (5k calls) | Testing, not production | | **Quiver Free** | 500 calls/mo | $30/mo (5k calls) | Testing, not production |
| **FMP Free** | 250 calls/day | $15/mo | Alternative for trades | | **FMP Free** | 250 calls/day | $15/mo | Alternative for trades |
@@ -222,5 +222,5 @@ class HouseWatcherClient:
} }
``` ```
Let me know if you want me to implement this scraper now for PR2! 🚀 Let me know if you want me to implement this scraper now for PR2!
+2 -2
View File
@@ -4,14 +4,14 @@
POTE can be deployed in several ways depending on your needs: POTE can be deployed in several ways depending on your needs:
1. **Local Development** (SQLite) - What you have now 1. **Local Development** (SQLite) - What you have now
2. **Single Server** (PostgreSQL + cron jobs) 2. **Single Server** (PostgreSQL + cron jobs)
3. **Docker** (Containerized, easy to move) 3. **Docker** (Containerized, easy to move)
4. **Cloud** (AWS/GCP/Azure with managed DB) 4. **Cloud** (AWS/GCP/Azure with managed DB)
--- ---
## Option 1: Local Development (Current Setup) ## Option 1: Local Development (Current Setup)
**You're already running this!** **You're already running this!**
+16 -16
View File
@@ -2,17 +2,17 @@
## Why Proxmox is Perfect for POTE ## Why Proxmox is Perfect for POTE
**Full control** - Your hardware, your rules **Full control** - Your hardware, your rules
**No monthly costs** - Just electricity **No monthly costs** - Just electricity
**Isolated VMs/LXC** - Clean environments **Isolated VMs/LXC** - Clean environments
**Snapshots** - Easy rollback if needed **Snapshots** - Easy rollback if needed
**Resource efficient** - Run alongside other services **Resource efficient** - Run alongside other services
--- ---
## Deployment Options on Proxmox ## Deployment Options on Proxmox
### Option 1: LXC Container (Recommended) ### Option 1: LXC Container (Recommended)
**Pros**: Lightweight, fast, efficient resource usage **Pros**: Lightweight, fast, efficient resource usage
**Cons**: Linux only (fine for POTE) **Cons**: Linux only (fine for POTE)
@@ -501,14 +501,14 @@ vs.
## Next Steps ## Next Steps
1. Create LXC container 1. Create LXC container
2. Install dependencies 2. Install dependencies
3. Setup PostgreSQL 3. Setup PostgreSQL
4. Deploy POTE 4. Deploy POTE
5. Configure cron jobs 5. Configure cron jobs
6. Setup backups 6. Setup backups
7. Build Phase 2 (Analytics) 7. ⏭ Build Phase 2 (Analytics)
8. Add FastAPI dashboard (optional) 8. ⏭ Add FastAPI dashboard (optional)
--- ---
@@ -583,7 +583,7 @@ EOF
sudo -u poteapp mkdir -p /home/poteapp/logs sudo -u poteapp mkdir -p /home/poteapp/logs
echo "" echo ""
echo " Setup complete!" echo " Setup complete!"
echo "" echo ""
echo "Next steps:" echo "Next steps:"
echo "1. su - poteapp" echo "1. su - poteapp"
@@ -600,5 +600,5 @@ chmod +x proxmox_setup.sh
--- ---
**Your Proxmox setup gives you enterprise-grade infrastructure at hobby costs!** 🚀 **Your Proxmox setup gives you enterprise-grade infrastructure at hobby costs!**
+5 -5
View File
@@ -137,13 +137,13 @@ python scripts/fetch_sample_prices.py
## Data Sources ## Data Sources
### Currently Working: ### Currently Working:
- yfinance (prices, company info) - yfinance (prices, company info)
- Manual entry - Manual entry
- CSV import - CSV import
- Fixture files (testing) - Fixture files (testing)
### Currently Down: ### Currently Down:
- House Stock Watcher API (domain issues) - House Stock Watcher API (domain issues)
### Future Options: ### Future Options:
- QuiverQuant (requires $30/month subscription) - QuiverQuant (requires $30/month subscription)
+19 -19
View File
@@ -8,10 +8,10 @@
### **Reality Check: No Real-Time Data Exists** ### **Reality Check: No Real-Time Data Exists**
**Federal Law (STOCK Act):** **Federal Law (STOCK Act):**
- 📅 Congress members have **30-45 days** to disclose trades - Congress members have **30-45 days** to disclose trades
- 📅 Disclosures are filed as **Periodic Transaction Reports (PTRs)** - Disclosures are filed as **Periodic Transaction Reports (PTRs)**
- 📅 Public databases update **after** filing (usually next day) - Public databases update **after** filing (usually next day)
- 📅 **No real-time feed exists by design** - **No real-time feed exists by design**
**Example Timeline:** **Example Timeline:**
``` ```
@@ -25,15 +25,15 @@ Feb 17, 2024 → Your system fetches it
Since trades appear in batches (not continuously), **running once per day is optimal**: Since trades appear in batches (not continuously), **running once per day is optimal**:
**Daily (7 AM)** - Catches overnight filings **Daily (7 AM)** - Catches overnight filings
**After market close** - Prices are final **After market close** - Prices are final
**Low server load** - Off-peak hours **Low server load** - Off-peak hours
**Hourly** - Wasteful, no new data **Hourly** - Wasteful, no new data
**Real-time** - Impossible, not how disclosures work **Real-time** - Impossible, not how disclosures work
--- ---
## 🤖 Automated Setup Options ## Automated Setup Options
### **Option 1: Cron Job (Linux/Proxmox) - Recommended** ### **Option 1: Cron Job (Linux/Proxmox) - Recommended**
@@ -149,7 +149,7 @@ Or from anywhere:
--- ---
## 📊 What Gets Updated? ## What Gets Updated?
### **1. Congressional Trades** ### **1. Congressional Trades**
**Script:** `fetch_congressional_trades.py` **Script:** `fetch_congressional_trades.py`
@@ -182,7 +182,7 @@ Or from anywhere:
--- ---
## ⚙️ Customizing the Schedule ## Customizing the Schedule
### **Different Frequencies** ### **Different Frequencies**
@@ -215,7 +215,7 @@ Or from anywhere:
--- ---
## 📧 Email Notifications (Optional) ## Email Notifications (Optional)
### **Setup Email Alerts** ### **Setup Email Alerts**
@@ -310,7 +310,7 @@ python scripts/email_summary.py
--- ---
## 🔍 Monitoring & Logging ## Monitoring & Logging
### **Check Cron Job Status** ### **Check Cron Job Status**
@@ -355,7 +355,7 @@ Add to `/etc/logrotate.d/pote`:
--- ---
## 🚨 Handling Failures ## Handling Failures
### **What If House Stock Watcher Is Down?** ### **What If House Stock Watcher Is Down?**
@@ -363,7 +363,7 @@ The script is designed to continue even if one step fails:
```bash ```bash
# Script continues and logs warnings # Script continues and logs warnings
⚠️ WARNING: Failed to fetch congressional trades WARNING: Failed to fetch congressional trades
This is likely because House Stock Watcher API is down This is likely because House Stock Watcher API is down
Continuing with other steps... Continuing with other steps...
``` ```
@@ -399,7 +399,7 @@ for attempt in range(MAX_RETRIES):
--- ---
## 📈 Performance Optimization ## Performance Optimization
### **Batch Processing** ### **Batch Processing**
@@ -439,7 +439,7 @@ CREATE INDEX IF NOT EXISTS ix_prices_security_id ON prices(security_id);
--- ---
## 🎯 Recommended Setup ## Recommended Setup
### **For Proxmox Production:** ### **For Proxmox Production:**
@@ -475,7 +475,7 @@ pote-update
--- ---
## 📝 Summary ## Summary
### **Key Points:** ### **Key Points:**
+67 -67
View File
@@ -1,13 +1,13 @@
# Live Market Monitoring + Congressional Trading Analysis # Live Market Monitoring + Congressional Trading Analysis
## 🎯 What's Possible vs Impossible ## What's Possible vs Impossible
### **NOT Possible:** ### **NOT Possible:**
- Identify WHO is buying/selling in real-time - Identify WHO is buying/selling in real-time
- Match live trades to specific Congress members - Match live trades to specific Congress members
- See congressional trades before they're disclosed - See congressional trades before they're disclosed
### **IS Possible:** ### **IS Possible:**
- Track unusual market activity in real-time - Track unusual market activity in real-time
- Monitor stocks Congress members historically trade - Monitor stocks Congress members historically trade
- Compare unusual activity to later disclosures - Compare unusual activity to later disclosures
@@ -15,7 +15,7 @@
--- ---
## 🔄 **Two-Phase Monitoring System** ## **Two-Phase Monitoring System**
### **Phase 1: Real-Time Market Monitoring** ### **Phase 1: Real-Time Market Monitoring**
Monitor unusual activity in stocks Congress trades: Monitor unusual activity in stocks Congress trades:
@@ -33,7 +33,7 @@ When disclosures come in:
--- ---
## 📊 **Implementation: Watchlist-Based Monitoring** ## **Implementation: Watchlist-Based Monitoring**
### **Concept:** ### **Concept:**
@@ -57,38 +57,38 @@ Step 3: When Disclosure Appears (30-45 days later)
--- ---
## 🛠️ **Data Sources for Live Market Monitoring** ## **Data Sources for Live Market Monitoring**
### **Free/Low-Cost Options:** ### **Free/Low-Cost Options:**
1. **Yahoo Finance (yfinance)** 1. **Yahoo Finance (yfinance)**
- Real-time quotes (15-min delay free) - Real-time quotes (15-min delay free)
- Historical options data - Historical options data
- Volume data - Volume data
- Not true real-time for options flow - Not true real-time for options flow
2. **Unusual Whales API** 2. **Unusual Whales API**
- Options flow data - Options flow data
- Unusual activity alerts - Unusual activity alerts
- 💰 Paid ($50-200/month) - Paid ($50-200/month)
- https://unusualwhales.com/ - https://unusualwhales.com/
3. **Tradier API** 3. **Tradier API**
- Real-time market data - Real-time market data
- Options chains - Options chains
- 💰 Paid but affordable ($10-50/month) - Paid but affordable ($10-50/month)
- https://tradier.com/ - https://tradier.com/
4. **FlowAlgo** 4. **FlowAlgo**
- Options flow tracking - Options flow tracking
- Dark pool data - Dark pool data
- 💰 Paid ($99-399/month) - Paid ($99-399/month)
- https://www.flowalgo.com/ - https://www.flowalgo.com/
5. **Polygon.io** 5. **Polygon.io**
- Real-time stock data - Real-time stock data
- Options data - Options data
- 💰 Free tier + paid plans - Free tier + paid plans
- https://polygon.io/ - https://polygon.io/
### **Best Free Option: Build Your Own with yfinance** ### **Best Free Option: Build Your Own with yfinance**
@@ -97,7 +97,7 @@ Track volume/price changes every 5 minutes for congressional watchlist tickers.
--- ---
## 💡 **Practical Hybrid System** ## **Practical Hybrid System**
### **What We Can Build:** ### **What We Can Build:**
@@ -141,37 +141,37 @@ for disclosure in new_disclosures:
--- ---
## 📈 **Example: Nancy Pelosi NVDA Trade Analysis** ## **Example: Nancy Pelosi NVDA Trade Analysis**
### **Timeline:** ### **Timeline:**
``` ```
Nov 10, 2024: Nov 10, 2024:
🔔 ALERT: NVDA unusual call options activity ALERT: NVDA unusual call options activity
Volume: 10x average Volume: 10x average
Strike: $500 (2 weeks out) Strike: $500 (2 weeks out)
Nov 15, 2024: Nov 15, 2024:
💰 Someone buys NVDA (unknown who at the time) Someone buys NVDA (unknown who at the time)
Nov 18, 2024: Nov 18, 2024:
📰 NVDA announces new AI chip NVDA announces new AI chip
📈 Stock jumps 15% Stock jumps 15%
Dec 15, 2024: Dec 15, 2024:
📋 Disclosure: Nancy Pelosi bought NVDA on Nov 15 Disclosure: Nancy Pelosi bought NVDA on Nov 15
Value: $15,001-$50,000 Value: $15,001-$50,000
ANALYSIS: ANALYSIS:
She bought AFTER unusual options activity (Nov 10) She bought AFTER unusual options activity (Nov 10)
She bought BEFORE announcement (Nov 18) She bought BEFORE announcement (Nov 18)
Timing: 3 days before major news ⏱ Timing: 3 days before major news
🚩 Flag: Investigate if announcement was public knowledge Flag: Investigate if announcement was public knowledge
``` ```
--- ---
## 🎯 **Recommended Approach** ## **Recommended Approach**
### **Phase 1: Build Congressional Ticker Watchlist** ### **Phase 1: Build Congressional Ticker Watchlist**
@@ -230,12 +230,12 @@ def monitor_tickers(tickers, interval_minutes=5):
# Check for unusual volume # Check for unusual volume
avg_volume = current['Volume'].mean() avg_volume = current['Volume'].mean()
if latest['Volume'] > avg_volume * 3: if latest['Volume'] > avg_volume * 3:
alert(f"🔔 {ticker}: Unusual volume spike!") alert(f" {ticker}: Unusual volume spike!")
# Check for price movement # Check for price movement
price_change = (latest['Close'] - current['Open'].iloc[0]) / current['Open'].iloc[0] price_change = (latest['Close'] - current['Open'].iloc[0]) / current['Open'].iloc[0]
if abs(price_change) > 0.05: # 5% move if abs(price_change) > 0.05: # 5% move
alert(f"📈 {ticker}: {price_change:.2%} move today!") alert(f" {ticker}: {price_change:.2%} move today!")
except Exception as e: except Exception as e:
print(f"Error monitoring {ticker}: {e}") print(f"Error monitoring {ticker}: {e}")
@@ -290,31 +290,31 @@ def analyze_disclosure_timing(disclosure):
--- ---
## 🚨 **Realistic Expectations** ## **Realistic Expectations**
### **What This System Will Do:** ### **What This System Will Do:**
Monitor stocks Congress members historically trade Monitor stocks Congress members historically trade
Alert on unusual market activity in those stocks Alert on unusual market activity in those stocks
Retroactively correlate disclosures with earlier alerts Retroactively correlate disclosures with earlier alerts
Identify timing patterns and potential advantages Identify timing patterns and potential advantages
Build database of congressional trading patterns Build database of congressional trading patterns
### **What This System WON'T Do:** ### **What This System WON'T Do:**
Identify WHO is buying in real-time Identify WHO is buying in real-time
Give you advance notice of congressional trades Give you advance notice of congressional trades
Provide real-time inside information Provide real-time inside information
Allow you to "front-run" Congress Allow you to "front-run" Congress
### **Legal & Ethical:** ### **Legal & Ethical:**
All data is public All data is public
Analysis is retrospective Analysis is retrospective
For research and transparency For research and transparency
Not market manipulation Not market manipulation
Cannot and should not be used to replicate potentially illegal trades Cannot and should not be used to replicate potentially illegal trades
--- ---
## 📊 **Proposed Implementation** ## **Proposed Implementation**
### **New Scripts to Create:** ### **New Scripts to Create:**
@@ -365,15 +365,15 @@ CREATE TABLE disclosure_timing_analysis (
--- ---
## 🎯 **Summary** ## **Summary**
### **Your Question:** ### **Your Question:**
> "Can we read live trades being made and compare them to a name?" > "Can we read live trades being made and compare them to a name?"
### **Answer:** ### **Answer:**
**No** - Live trades are anonymous, can't identify individuals **No** - Live trades are anonymous, can't identify individuals
**BUT** - You CAN: **BUT** - You CAN:
1. Monitor unusual activity in stocks Congress trades 1. Monitor unusual activity in stocks Congress trades
2. Log these alerts in real-time 2. Log these alerts in real-time
3. When disclosures appear (30-45 days later), correlate them 3. When disclosures appear (30-45 days later), correlate them
@@ -381,25 +381,25 @@ CREATE TABLE disclosure_timing_analysis (
5. Build patterns database of timing and performance 5. Build patterns database of timing and performance
### **This Gives You:** ### **This Gives You:**
- Transparency on timing advantages - Transparency on timing advantages
- Pattern detection across officials - Pattern detection across officials
- Research-grade analysis - Research-grade analysis
- Historical correlation data - Historical correlation data
### **This Does NOT Give You:** ### **This Does NOT Give You:**
- Real-time identity of traders - Real-time identity of traders
- Advance notice of congressional trades - Advance notice of congressional trades
- Ability to "front-run" disclosures - Ability to "front-run" disclosures
--- ---
## 🚀 **Would You Like Me To Build This?** ## **Would You Like Me To Build This?**
I can create: I can create:
1. Real-time monitoring system for congressional tickers 1. Real-time monitoring system for congressional tickers
2. Alert logging and analysis 2. Alert logging and analysis
3. Timing correlation when disclosures appear 3. Timing correlation when disclosures appear
4. Pattern detection and reporting 4. Pattern detection and reporting
This would be **Phase 2.5** of POTE - the "timing analysis" module. This would be **Phase 2.5** of POTE - the "timing analysis" module.
+5 -5
View File
@@ -205,18 +205,18 @@ Output:
POTE HEALTH CHECK POTE HEALTH CHECK
============================================================ ============================================================
Timestamp: 2025-12-15T10:30:00 Timestamp: 2025-12-15T10:30:00
Overall Status: OK Overall Status: OK
Database Connection: Database connection successful Database Connection: Database connection successful
Data Freshness: Data is fresh (2 days old) Data Freshness: Data is fresh (2 days old)
latest_trade_date: 2025-12-13 latest_trade_date: 2025-12-13
Data Counts: Database has 1,234 trades Data Counts: Database has 1,234 trades
officials: 45 officials: 45
securities: 123 securities: 123
trades: 1,234 trades: 1,234
prices: 12,345 prices: 12,345
market_alerts: 567 market_alerts: 567
Recent Alerts: 23 alerts in last 24 hours Recent Alerts: 23 alerts in last 24 hours
============================================================ ============================================================
``` ```
+35 -35
View File
@@ -22,18 +22,18 @@ chmod 600 .env
chown poteapp:poteapp .env chown poteapp:poteapp .env
``` ```
### Pros ### Pros
- Simple, works immediately - Simple, works immediately
- No additional setup - No additional setup
- Standard practice for Python projects - Standard practice for Python projects
### ⚠️ Cons ### Cons
- Secrets stored in plain text on disk - Secrets stored in plain text on disk
- Risk if server is compromised - Risk if server is compromised
- No audit trail - No audit trail
### 🔒 Security Checklist ### Security Checklist
- [ ] `.env` in `.gitignore` (already done ) - [ ] `.env` in `.gitignore` (already done )
- [ ] File permissions: `chmod 600 .env` - [ ] File permissions: `chmod 600 .env`
- [ ] Never commit to git - [ ] Never commit to git
- [ ] Backup securely (encrypted) - [ ] Backup securely (encrypted)
@@ -76,12 +76,12 @@ sudo chmod 600 /etc/systemd/system/pote.service
sudo systemctl daemon-reload sudo systemctl daemon-reload
``` ```
### Pros ### Pros
- Secrets not in git or project directory - Secrets not in git or project directory
- Standard Linux practice - Standard Linux practice
- Works with systemd timers - Works with systemd timers
### ⚠️ Cons ### Cons
- Still visible in `systemctl show` - Still visible in `systemctl show`
- Requires root to edit - Requires root to edit
@@ -128,12 +128,12 @@ source .env
python scripts/send_daily_report.py python scripts/send_daily_report.py
``` ```
### Pros ### Pros
- Secrets separate from code - Secrets separate from code
- Easy to rotate - Easy to rotate
- Can be backed up separately - Can be backed up separately
### ⚠️ Cons ### Cons
- Extra file to manage - Extra file to manage
- Still plain text - Still plain text
@@ -191,12 +191,12 @@ class Settings(BaseSettings):
smtp_password: str = Field(default_factory=lambda: get_secret("SMTP_PASSWORD")) smtp_password: str = Field(default_factory=lambda: get_secret("SMTP_PASSWORD"))
``` ```
### Pros ### Pros
- Docker-native solution - Docker-native solution
- Encrypted in Swarm mode - Encrypted in Swarm mode
- Never in logs - Never in logs
### ⚠️ Cons ### Cons
- Requires Docker - Requires Docker
- More complex setup - More complex setup
@@ -228,13 +228,13 @@ secrets = client.secrets.kv.v2.read_secret_version(path='pote')
smtp_password = secrets['data']['data']['smtp_password'] smtp_password = secrets['data']['data']['smtp_password']
``` ```
### Pros ### Pros
- Centralized secrets management - Centralized secrets management
- Audit logs - Audit logs
- Dynamic secrets - Dynamic secrets
- Access control - Access control
### ⚠️ Cons ### Cons
- Complex setup - Complex setup
- Requires Vault infrastructure - Requires Vault infrastructure
- Overkill for single user - Overkill for single user
@@ -260,14 +260,14 @@ env:
DATABASE_URL: postgresql://user:${{ secrets.DB_PASSWORD }}@postgres/db DATABASE_URL: postgresql://user:${{ secrets.DB_PASSWORD }}@postgres/db
``` ```
### ⚠️ Important ### Important
- **Only for CI/CD pipelines** - **Only for CI/CD pipelines**
- **NOT for deployed servers** - **NOT for deployed servers**
- Secrets are injected during workflow runs - Secrets are injected during workflow runs
--- ---
## 🎯 Recommendation for Your Setup ## Recommendation for Your Setup
### Personal/Research Use (Current) ### Personal/Research Use (Current)
@@ -305,9 +305,9 @@ gpg -c .env # Creates .env.gpg
--- ---
## 🔒 General Security Best Practices ## General Security Best Practices
### DO ### DO
- Use strong, unique passwords - Use strong, unique passwords
- Restrict file permissions (`chmod 600`) - Restrict file permissions (`chmod 600`)
@@ -316,7 +316,7 @@ gpg -c .env # Creates .env.gpg
- Use encrypted backups - Use encrypted backups
- Audit who has server access - Audit who has server access
### DON'T ### DON'T
- Commit secrets to git (even private repos) - Commit secrets to git (even private repos)
- Store passwords in code - Store passwords in code
@@ -327,7 +327,7 @@ gpg -c .env # Creates .env.gpg
--- ---
## 🧪 Test Your Security ## Test Your Security
### Check if `.env` is protected ### Check if `.env` is protected
@@ -352,7 +352,7 @@ git log --all --full-history --source --pickaxe-all -S 'smtp_password'
--- ---
## 🔄 Password Rotation Procedure ## Password Rotation Procedure
### Every 90 days (or if compromised): ### Every 90 days (or if compromised):
@@ -369,32 +369,32 @@ git log --all --full-history --source --pickaxe-all -S 'smtp_password'
--- ---
## 📊 Security Level Comparison ## Security Level Comparison
| Level | Method | Effort | Protection | | Level | Method | Effort | Protection |
|-------|--------|--------|------------| |-------|--------|--------|------------|
| 🔓 Basic | `.env` (default perms) | None | Low | | Basic | `.env` (default perms) | None | Low |
| 🔒 Good | `.env` (chmod 600) | 1 min | Medium | | Good | `.env` (chmod 600) | 1 min | Medium |
| 🔒 Better | Environment variables | 10 min | Good | | Better | Environment variables | 10 min | Good |
| 🔒 Better | Separate secrets file | 10 min | Good | | Better | Separate secrets file | 10 min | Good |
| 🔐 Best | Docker Secrets | 30 min | Very Good | | Best | Docker Secrets | 30 min | Very Good |
| 🔐 Best | Vault | 2+ hours | Excellent | | Best | Vault | 2+ hours | Excellent |
--- ---
## 🎯 Your Current Status ## Your Current Status
**Already secure enough for personal use:** **Already secure enough for personal use:**
- `.env` in `.gitignore` - `.env` in `.gitignore`
- Not committed to git - Not committed to git
- Local server only - Local server only
⚠️ **Recommended improvement (2 minutes):** **Recommended improvement (2 minutes):**
```bash ```bash
chmod 600 .env chmod 600 .env
``` ```
🔐 **Optional (if paranoid):** **Optional (if paranoid):**
- Use separate secrets file in `/etc/pote/` - Use separate secrets file in `/etc/pote/`
- Encrypt backups with GPG - Encrypt backups with GPG
- Set up password rotation schedule - Set up password rotation schedule
@@ -405,9 +405,9 @@ chmod 600 .env
**For your levkin.ca setup:** **For your levkin.ca setup:**
1. **Current approach (`.env` file) is fine** 1. **Current approach (`.env` file) is fine**
2. **Add `chmod 600 .env`** for better security (2 minutes) 2. **Add `chmod 600 .env`** for better security (2 minutes)
3. **Don't commit `.env` to git** (already protected ) 3. **Don't commit `.env` to git** (already protected )
4. **Consider upgrading to environment variables** if you deploy to production 4. **Consider upgrading to environment variables** if you deploy to production
Your current setup is **appropriate for a personal research project**. Don't over-engineer it unless you have specific compliance requirements or a team. Your current setup is **appropriate for a personal research project**. Don't over-engineer it unless you have specific compliance requirements or a team.
@@ -62,7 +62,7 @@ Follow the prompts:
2. Choose daily report time (recommend 6 AM) 2. Choose daily report time (recommend 6 AM)
3. Confirm 3. Confirm
That's it! 🎉 That's it!
--- ---
@@ -112,7 +112,7 @@ Run the daily script manually to test:
./scripts/automated_daily_run.sh ./scripts/automated_daily_run.sh
``` ```
Check if email arrived! 📧 Check if email arrived!
--- ---
@@ -244,5 +244,5 @@ Add to cron for regular health checks:
--- ---
**You're all set! POTE will now run automatically and send you daily/weekly reports. 🚀** **You're all set! POTE will now run automatically and send you daily/weekly reports. **
@@ -1,17 +1,17 @@
# POTE Deployment & Automation Guide # POTE Deployment & Automation Guide
## 🎯 Quick Answer to Your Questions ## Quick Answer to Your Questions
### After Deployment, What Happens? ### After Deployment, What Happens?
**By default: NOTHING automatic happens.** You need to set up automation. **By default: NOTHING automatic happens.** You need to set up automation.
The deployed system is: The deployed system is:
- Running (database, code installed) - Running (database, code installed)
- Accessible via SSH at your Proxmox IP - Accessible via SSH at your Proxmox IP
- NOT fetching data automatically - NOT fetching data automatically
- NOT sending reports automatically - NOT sending reports automatically
- NOT monitoring markets automatically - NOT monitoring markets automatically
**You must either:** **You must either:**
1. **Run scripts manually** when you want updates, OR 1. **Run scripts manually** when you want updates, OR
@@ -19,7 +19,7 @@ The deployed system is:
--- ---
## 🚀 Option 1: Automated Email Reports (Recommended) ## Option 1: Automated Email Reports (Recommended)
### What You Get ### What You Get
@@ -53,13 +53,13 @@ Run the interactive setup:
Follow prompts: Follow prompts:
1. Enter your email address 1. Enter your email address
2. Choose report time (default: 6 AM) 2. Choose report time (default: 6 AM)
3. Done! 3. Done!
**See full guide:** [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md) **See full guide:** [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md)
--- ---
## 📍 Option 2: Access Reports via IP (No Email) ## Option 2: Access Reports via IP (No Email)
If you don't want email, you can: If you don't want email, you can:
@@ -107,7 +107,7 @@ python scripts/health_check.py
--- ---
## 🌐 Option 3: Build a Web Interface (Future) ## Option 3: Build a Web Interface (Future)
Currently, POTE is **command-line only**. No web UI yet. Currently, POTE is **command-line only**. No web UI yet.
@@ -122,16 +122,16 @@ For now, use SSH or email reports.
--- ---
## 🔄 Do You Need CI/CD Pipelines? ## Do You Need CI/CD Pipelines?
### What the Pipeline Does ### What the Pipeline Does
The included CI/CD pipeline (`.github/workflows/ci.yml`) runs on **every git push**: The included CI/CD pipeline (`.github/workflows/ci.yml`) runs on **every git push**:
1. Lint & test (93 tests) 1. Lint & test (93 tests)
2. Security scanning 2. Security scanning
3. Dependency scanning 3. Dependency scanning
4. Docker build test 4. Docker build test
### Should You Use It? ### Should You Use It?
@@ -174,18 +174,18 @@ docker build -t pote:test .
--- ---
## 📊 Comparison of Options ## Comparison of Options
| Method | Pros | Cons | Best For | | Method | Pros | Cons | Best For |
|--------|------|------|----------| |--------|------|------|----------|
| **Automated Email** | Convenient<br> No SSH needed<br> Daily/weekly updates | Requires SMTP setup | Most users | | **Automated Email** | Convenient<br> No SSH needed<br> Daily/weekly updates | Requires SMTP setup | Most users |
| **SSH + Manual Scripts** | Full control<br> No email needed | Manual work<br> Must remember to run | Power users | | **SSH + Manual Scripts** | Full control<br> No email needed | Manual work<br> Must remember to run | Power users |
| **Saved Reports (SSH access)** | Automated<br> No email | Must SSH to view | Users without email | | **Saved Reports (SSH access)** | Automated<br> No email | Must SSH to view | Users without email |
| **Web Interface** | User-friendly | Not implemented yet | Future | | **Web Interface** | User-friendly | Not implemented yet | Future |
--- ---
## 🛠️ Your Ansible Pipeline ## Your Ansible Pipeline
Your existing Ansible CI/CD pipeline is **NOT directly usable** for POTE because: Your existing Ansible CI/CD pipeline is **NOT directly usable** for POTE because:
@@ -197,11 +197,11 @@ Your existing Ansible CI/CD pipeline is **NOT directly usable** for POTE because
**Concepts from your pipeline that ARE used in POTE's CI/CD:** **Concepts from your pipeline that ARE used in POTE's CI/CD:**
- Security scanning (Trivy, Bandit instead of Gitleaks) - Security scanning (Trivy, Bandit instead of Gitleaks)
- Dependency scanning (Trivy instead of npm audit) - Dependency scanning (Trivy instead of npm audit)
- SAST scanning (Bandit instead of Semgrep) - SAST scanning (Bandit instead of Semgrep)
- Container scanning (Docker build test) - Container scanning (Docker build test)
- Workflow summary generation - Workflow summary generation
**The POTE pipeline (`.github/workflows/ci.yml`) already includes all of these!** **The POTE pipeline (`.github/workflows/ci.yml`) already includes all of these!**
@@ -217,7 +217,7 @@ Your Gitea server can run the POTE pipeline using Gitea Actions:
--- ---
## 📧 Email Setup Examples ## Email Setup Examples
### Gmail (Most Common) ### Gmail (Most Common)
@@ -258,7 +258,7 @@ REPORT_RECIPIENTS=admin@yourdomain.com
--- ---
## Recommended Setup for Most Users ## Recommended Setup for Most Users
1. **Deploy to Proxmox** (5 min) 1. **Deploy to Proxmox** (5 min)
```bash ```bash
@@ -283,7 +283,7 @@ REPORT_RECIPIENTS=admin@yourdomain.com
--- ---
## 🔍 Monitoring & Health Checks ## Monitoring & Health Checks
### Add System Health Monitoring ### Add System Health Monitoring
@@ -310,17 +310,17 @@ python scripts/health_check.py
--- ---
## 📚 Full Documentation ## Full Documentation
- **Automation Setup**: [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md) - **Automation Setup**: [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md)
- **Deployment**: [`PROXMOX_QUICKSTART.md`](PROXMOX_QUICKSTART.md) - **Deployment**: [`PROXMOX_QUICKSTART.md`](PROXMOX_QUICKSTART.md)
- **Usage**: [`QUICKSTART.md`](QUICKSTART.md) - **Usage**: [`QUICKSTART.md`](QUICKSTART.md)
- **Detailed Automation Guide**: [`docs/12_automation_and_reporting.md`](docs/12_automation_and_reporting.md) - **Detailed Automation Guide**: [`docs/12_automation_and_reporting.md`](docs/12_automation_and_reporting.md)
- **Monitoring System**: [`MONITORING_SYSTEM_COMPLETE.md`](MONITORING_SYSTEM_COMPLETE.md) - **Monitoring System**: [`MONITORING_SYSTEM_COMPLETE.md`](MONITORING_SYSTEM_COMPLETE.md)
--- ---
## 🎉 Summary ## Summary
**After deployment:** **After deployment:**
- Reports are NOT sent automatically by default - Reports are NOT sent automatically by default
@@ -334,5 +334,5 @@ python scripts/health_check.py
1. Deploy to Proxmox 1. Deploy to Proxmox
2. Run `./scripts/setup_cron.sh` 2. Run `./scripts/setup_cron.sh`
3. Receive daily/weekly email reports 3. Receive daily/weekly email reports
4. Done! 🚀 4. Done!
+10 -10
View File
@@ -2,7 +2,7 @@
Homelab POTE sends via Mailcow **`mail.levkine.ca`** using the shared **`alerts@levkine.ca`** mailbox (same as Kuma/Beszel). See ansible `docs/guides/smtp-inventory.md`. Homelab POTE sends via Mailcow **`mail.levkine.ca`** using the shared **`alerts@levkine.ca`** mailbox (same as Kuma/Beszel). See ansible `docs/guides/smtp-inventory.md`.
## Configuration Done ## Configuration Done
The `.env` file has been created with these settings: The `.env` file has been created with these settings:
@@ -15,7 +15,7 @@ FROM_EMAIL=alerts@levkine.ca
REPORT_RECIPIENTS=idobkin@gmail.com REPORT_RECIPIENTS=idobkin@gmail.com
``` ```
## 🔑 Next Steps ## Next Steps
### 1. Add Your Password ### 1. Add Your Password
@@ -38,7 +38,7 @@ python scripts/send_daily_report.py --to test@levkin.ca --test-smtp
If successful, you'll see: If successful, you'll see:
``` ```
SMTP connection test successful! SMTP connection test successful!
Daily report sent successfully! Daily report sent successfully!
``` ```
And you should receive a test email at `test@levkin.ca`! And you should receive a test email at `test@levkin.ca`!
@@ -57,7 +57,7 @@ This will:
- Schedule daily reports (default: 6 AM) - Schedule daily reports (default: 6 AM)
- Schedule weekly reports (Sundays at 8 AM) - Schedule weekly reports (Sundays at 8 AM)
## 📧 Email Server Details (For Reference) ## Email Server Details (For Reference)
Based on your Thunderbird setup: Based on your Thunderbird setup:
@@ -76,10 +76,10 @@ Based on your Thunderbird setup:
POTE only uses **SMTP (outgoing)** to send reports. POTE only uses **SMTP (outgoing)** to send reports.
## 🔒 Security Notes ## Security Notes
1. **Never commit `.env` to git!** 1. **Never commit `.env` to git!**
- Already in `.gitignore` - Already in `.gitignore`
- Contains sensitive password - Contains sensitive password
2. **Password Security:** 2. **Password Security:**
@@ -92,7 +92,7 @@ POTE only uses **SMTP (outgoing)** to send reports.
chmod 600 .env # Only owner can read/write chmod 600 .env # Only owner can read/write
``` ```
## 🎯 Change Recipients ## Change Recipients
To send reports to different email addresses (not just test@levkin.ca): To send reports to different email addresses (not just test@levkin.ca):
@@ -109,14 +109,14 @@ python scripts/send_daily_report.py --to someone-else@example.com
# The FROM address will still be test@levkin.ca # The FROM address will still be test@levkin.ca
``` ```
## Testing Checklist ## Testing Checklist
- [ ] Updated `.env` with your actual password - [ ] Updated `.env` with your actual password
- [ ] Run `python scripts/send_daily_report.py --to test@levkin.ca --test-smtp` - [ ] Run `python scripts/send_daily_report.py --to test@levkin.ca --test-smtp`
- [ ] Checked inbox at test@levkin.ca (check spam folder!) - [ ] Checked inbox at test@levkin.ca (check spam folder!)
- [ ] If successful, run `./scripts/setup_cron.sh` to automate - [ ] If successful, run `./scripts/setup_cron.sh` to automate
## 🐛 Troubleshooting ## Troubleshooting
### Error: "SMTP connection failed" ### Error: "SMTP connection failed"
@@ -139,5 +139,5 @@ Check:
--- ---
**You're all set! POTE will send reports from test@levkin.ca 📧** **You're all set! POTE will send reports from test@levkin.ca **
@@ -2,14 +2,14 @@
## TL;DR: You can test everything for $0 ## TL;DR: You can test everything for $0
### Already Working (PR1 ) ### Already Working (PR1 )
- **Price data**: `yfinance` (free, unlimited) - **Price data**: `yfinance` (free, unlimited)
- **Unit tests**: Mocked data in `tests/` (15 passing tests) - **Unit tests**: Mocked data in `tests/` (15 passing tests)
- **Coverage**: 87% without any paid APIs - **Coverage**: 87% without any paid APIs
### For PR2 (Congressional Trades) - FREE Options ### For PR2 (Congressional Trades) - FREE Options
#### Best Option: House Stock Watcher 🌟 #### Best Option: House Stock Watcher
```bash ```bash
# No API key needed, just scrape their public JSON # No API key needed, just scrape their public JSON
curl https://housestockwatcher.com/api/all_transactions curl https://housestockwatcher.com/api/all_transactions
@@ -44,10 +44,10 @@ python scripts/fetch_house_watcher_sample.py # We'll build this in PR2
### What You DON'T Need to Pay For ### What You DON'T Need to Pay For
QuiverQuant Pro ($30/mo) - free tier is enough for dev/testing QuiverQuant Pro ($30/mo) - free tier is enough for dev/testing
Financial Modeling Prep paid tier - free tier works Financial Modeling Prep paid tier - free tier works
Any paid database hosting - SQLite works great locally Any paid database hosting - SQLite works great locally
Any cloud services - runs 100% locally Any cloud services - runs 100% locally
### When You MIGHT Want Paid (Way Later) ### When You MIGHT Want Paid (Way Later)
@@ -56,7 +56,7 @@ python scripts/fetch_house_watcher_sample.py # We'll build this in PR2
- Multiple concurrent users on a dashboard - Multiple concurrent users on a dashboard
- Commercial use (check each API's terms) - Commercial use (check each API's terms)
**For personal research? Stay free forever. 🎉** **For personal research? Stay free forever. **
--- ---
@@ -1,20 +1,20 @@
# 🔐 Gitea Secrets Guide for POTE # Gitea Secrets Guide for POTE
## YES! You Can Store Passwords in Gitea ## YES! You Can Store Passwords in Gitea
Gitea has a **Secrets** feature (like GitHub Actions secrets) that lets you store passwords securely and use them in: Gitea has a **Secrets** feature (like GitHub Actions secrets) that lets you store passwords securely and use them in:
1. **CI/CD pipelines** (Gitea Actions workflows) 1. **CI/CD pipelines** (Gitea Actions workflows)
2. **Deployment workflows** 2. **Deployment workflows**
**BUT NOT:** **BUT NOT:**
- Directly in your running application on Proxmox - Directly in your running application on Proxmox
- Accessed by scripts outside of workflows - Accessed by scripts outside of workflows
--- ---
## 🎯 What Gitea Secrets Are Good For ## What Gitea Secrets Are Good For
### Perfect Use Cases ### Perfect Use Cases
1. **CI/CD Testing** - Run tests with real credentials 1. **CI/CD Testing** - Run tests with real credentials
2. **Automated Deployment** - Deploy to Proxmox with SSH keys 2. **Automated Deployment** - Deploy to Proxmox with SSH keys
@@ -22,7 +22,7 @@ Gitea has a **Secrets** feature (like GitHub Actions secrets) that lets you stor
4. **Docker Registry** - Push images with credentials 4. **Docker Registry** - Push images with credentials
5. **API Keys** - Access external services during builds 5. **API Keys** - Access external services during builds
### NOT Good For ### NOT Good For
1. **Runtime secrets** - Your deployed app on Proxmox can't access them 1. **Runtime secrets** - Your deployed app on Proxmox can't access them
2. **Local development** - Can't use secrets on your laptop 2. **Local development** - Can't use secrets on your laptop
@@ -30,7 +30,7 @@ Gitea has a **Secrets** feature (like GitHub Actions secrets) that lets you stor
--- ---
## 🔧 How to Set Up Gitea Secrets ## How to Set Up Gitea Secrets
### Step 1: Add Secrets to Gitea ### Step 1: Add Secrets to Gitea
@@ -57,7 +57,7 @@ Secrets are accessed with `${{ secrets.SECRET_NAME }}` syntax.
--- ---
## 📝 Example: CI Pipeline with Secrets ## Example: CI Pipeline with Secrets
**File:** `.github/workflows/ci.yml` **File:** `.github/workflows/ci.yml`
@@ -92,11 +92,11 @@ jobs:
--smtp-password "${{ secrets.SMTP_PASSWORD }}" --smtp-password "${{ secrets.SMTP_PASSWORD }}"
``` ```
** I've already updated your CI pipeline to use secrets!** ** I've already updated your CI pipeline to use secrets!**
--- ---
## 🚀 Example: Automated Deployment Workflow ## Example: Automated Deployment Workflow
Create `.github/workflows/deploy.yml`: Create `.github/workflows/deploy.yml`:
@@ -152,7 +152,7 @@ jobs:
--- ---
## 🔄 How Secrets Flow to Your Server ## How Secrets Flow to Your Server
### Option 1: Deploy Workflow Updates `.env` (Recommended) ### Option 1: Deploy Workflow Updates `.env` (Recommended)
@@ -201,11 +201,11 @@ jobs:
--- ---
## 🎯 Recommended Setup for Your POTE Project ## Recommended Setup for Your POTE Project
### For CI/CD (Testing): ### For CI/CD (Testing):
**Use Gitea Secrets** **Use Gitea Secrets**
```yaml ```yaml
# .github/workflows/ci.yml (already updated!) # .github/workflows/ci.yml (already updated!)
@@ -216,7 +216,7 @@ env:
### For Deployed Server (Proxmox): ### For Deployed Server (Proxmox):
**Keep using `.env` file** **Keep using `.env` file**
Why? Why?
- Simpler for manual SSH access - Simpler for manual SSH access
@@ -227,7 +227,7 @@ Why?
--- ---
## 🚀 Complete Workflow: Gitea → Proxmox ## Complete Workflow: Gitea → Proxmox
### 1. Store Secrets in Gitea ### 1. Store Secrets in Gitea
@@ -264,27 +264,27 @@ git push origin main
--- ---
## ⚠️ Important Limitations ## Important Limitations
### Gitea Secrets CAN'T: ### Gitea Secrets CAN'T:
Be accessed outside of workflows Be accessed outside of workflows
Be used in local `python script.py` runs Be used in local `python script.py` runs
Be read by cron jobs on Proxmox (directly) Be read by cron jobs on Proxmox (directly)
Replace `.env` for runtime application config Replace `.env` for runtime application config
### Gitea Secrets CAN: ### Gitea Secrets CAN:
Secure your CI/CD pipeline Secure your CI/CD pipeline
Deploy safely without exposing passwords in git Deploy safely without exposing passwords in git
Update `.env` on server during deployment Update `.env` on server during deployment
Run automated tests with real credentials Run automated tests with real credentials
--- ---
## 🔒 Security Best Practices ## Security Best Practices
### DO: ### DO:
1. **Store ALL sensitive data as Gitea secrets** 1. **Store ALL sensitive data as Gitea secrets**
- SMTP passwords - SMTP passwords
@@ -300,10 +300,10 @@ git push origin main
3. **Never echo secrets** 3. **Never echo secrets**
```yaml ```yaml
# ❌ BAD - exposes in logs # BAD - exposes in logs
- run: echo "${{ secrets.PASSWORD }}" - run: echo "${{ secrets.PASSWORD }}"
# ✅ GOOD - masked automatically # GOOD - masked automatically
- run: use_password "${{ secrets.PASSWORD }}" - run: use_password "${{ secrets.PASSWORD }}"
``` ```
@@ -311,7 +311,7 @@ git push origin main
- Update in Gitea UI - Update in Gitea UI
- Re-run deployment workflow - Re-run deployment workflow
### DON'T: ### DON'T:
1. **Commit secrets to git** (even private repos) 1. **Commit secrets to git** (even private repos)
2. **Share secrets via Slack/email** 2. **Share secrets via Slack/email**
@@ -320,18 +320,18 @@ git push origin main
--- ---
## 📊 Comparison: Where to Store Secrets ## Comparison: Where to Store Secrets
| Storage | CI/CD | Deployed App | Easy Updates | Security | | Storage | CI/CD | Deployed App | Easy Updates | Security |
|---------|-------|--------------|--------------|----------| |---------|-------|--------------|--------------|----------|
| **Gitea Secrets** | Perfect | No | Via workflow | ⭐⭐⭐⭐⭐ | | **Gitea Secrets** | Perfect | No | Via workflow | |
| **`.env` file** | No | Perfect | `nano .env` | ⭐⭐⭐ | | **`.env` file** | No | Perfect | `nano .env` | |
| **Environment Vars** | Yes | Yes | Harder | ⭐⭐⭐⭐ | | **Environment Vars** | Yes | Yes | Harder | |
| **Both (Recommended)** | Yes | Yes | Automated | ⭐⭐⭐⭐⭐ | | **Both (Recommended)** | Yes | Yes | Automated | |
--- ---
## 🎯 My Recommendation for You ## My Recommendation for You
### Use BOTH: ### Use BOTH:
@@ -345,21 +345,21 @@ git push origin main
2. Commit code changes 2. Commit code changes
3. Push to Gitea 3. Push to Gitea
4. Workflow runs: 4. Workflow runs:
- Tests with Gitea secrets - Tests with Gitea secrets
- Deploys to Proxmox - Deploys to Proxmox
- Updates .env with secrets - Updates .env with secrets
5. Proxmox app reads from .env 5. Proxmox app reads from .env
``` ```
**This gives you:** **This gives you:**
- Secure CI/CD - Secure CI/CD
- Easy manual SSH access - Easy manual SSH access
- Automated deployments - Automated deployments
- No passwords in git - No passwords in git
--- ---
## 🚀 Next Steps ## Next Steps
### 1. Add Secrets to Gitea (5 minutes) ### 1. Add Secrets to Gitea (5 minutes)
@@ -387,7 +387,7 @@ I can create `.github/workflows/deploy.yml` if you want automated deployments!
--- ---
## 💡 Quick Commands ## Quick Commands
### Add SSH Key to Gitea (for deployment): ### Add SSH Key to Gitea (for deployment):
@@ -409,12 +409,12 @@ git commit --allow-empty -m "Test secrets"
git push git push
# Check Gitea Actions tab # Check Gitea Actions tab
# Look for green checkmarks # Look for green checkmarks
``` ```
--- ---
## 📚 See Also ## See Also
- **[docs/13_secrets_management.md](docs/13_secrets_management.md)** - All secrets options - **[docs/13_secrets_management.md](docs/13_secrets_management.md)** - All secrets options
- **[.github/workflows/ci.yml](.github/workflows/ci.yml)** - Updated with secrets support - **[.github/workflows/ci.yml](.github/workflows/ci.yml)** - Updated with secrets support
@@ -422,16 +422,16 @@ git push
--- ---
## Summary ## Summary
**YES, use Gitea secrets!** They're perfect for: **YES, use Gitea secrets!** They're perfect for:
- CI/CD pipelines - CI/CD pipelines
- Automated deployments - Automated deployments
- Keeping passwords out of git - Keeping passwords out of git
**But ALSO keep `.env` on Proxmox** for: **But ALSO keep `.env` on Proxmox** for:
- Runtime application config - Runtime application config
- Manual SSH access - Manual SSH access
- Cron jobs - Cron jobs
**Best of both worlds:** Gitea secrets deploy and update the `.env` file automatically! 🚀 **Best of both worlds:** Gitea secrets deploy and update the `.env` file automatically!
+163
View File
@@ -0,0 +1,163 @@
# POTE homelab handoff — 2026-05-27
**Status:** Production LXC running; PR #1 merged to `main`; CI green on Gitea Actions.
**Research only — not investment advice.**
---
## Whats live
| Item | Value |
|------|--------|
| Host | LXC **236** `pote` @ **10.0.10.48** (pve10) |
| App | `/home/poteapp/pote` (venv, **no git clone** — deploy via rsync) |
| DB | PostgreSQL `pote` / `poteuser` (password rotated; in Ansible vault) |
| Data | ~55 officials, ~329 trades (30-day live ingest, May 2026) |
| SMTP | `10.0.10.132` (Mailcow), send as **`alerts@levkine.ca`** |
| Reports | **`idobkin@gmail.com`** daily 07:00, weekly Sun 08:00 |
### Cron (`crontab -u poteapp -l`)
| Time | Script |
|------|--------|
| 06:00 | `fetch_congressional_trades.py --days 7` |
| 06:15 | `enrich_securities.py` |
| 06:30 | `monitor_market.py --scan` |
| 07:00 | `send_daily_report.py --to idobkin@gmail.com` |
| Sun 08:00 | `send_weekly_report.py --to idobkin@gmail.com` |
### Data source (important)
Legacy **housestockwatcher.com** and S3 buckets are dead/blocked. Ingest uses public JSON from [congress-trading-monitor](https://github.com/kadoa-org/congress-trading-monitor) (~5000 rows cap). Override with env `POTE_HOUSE_DATA_URL` if you add another feed.
---
## Repos & branches
| Repo | Branch | Notes |
|------|--------|--------|
| **POTE** | `main` @ `git.levkin.ca/ilia/POTE` | Merged PR #1 — ingest, email, CI, deps |
| **ansible** | `feature/outline-setup-api` (or `master`) | Inventory, `deploy-pote.sh`, vault — may need merge to homelab default branch |
Local:
```bash
cd ~/Documents/code/POTE && git checkout main && git pull
```
---
## Quick access
```bash
ssh root@10.0.10.48
su - poteapp
cd pote && source venv/bin/activate
# Logs
tail -f ~/logs/daily_report.log
tail -f ~/logs/trades.log
# Manual run
python scripts/fetch_congressional_trades.py --days 30
python scripts/send_daily_report.py --to idobkin@gmail.com --test-smtp
```
Deploy code from laptop (preserves server `.env`):
```bash
cd ~/Documents/code/ansible
make deploy-pote
# or: RUN_FETCH=1 make deploy-pote
```
---
## Ansible / homelab inventory
Already wired (ansible repo):
- `inventories/production/hosts``pote` @ `.48`, VMID 236
- `docs/guides/host-list.md` — LXC 236 row
- `scripts/beszel-install-agents.sh``pote-236`
- `scripts/deploy-pote.sh`, `make deploy-pote`
- `scripts/vault-update-pote.py`, `make vault-update-pote`
- `docs/guides/smtp-inventory.md` — POTE uses `alerts@levkine.ca`
- Vault: `vault_pote_db_password_prod`, `vault_pote_smtp_password`
```bash
make vault-export-env
make beszel-install-agents BESZEL_ONLY=pote-236 # if agent not yet installed
```
---
## Verify after first automated day
1. **07:00+** — Email in Gmail (From: `alerts@levkine.ca`, subject `POTE Daily Report - YYYY-MM-DD`). Check spam once.
2. **Logs**`~/logs/daily_report.log`, `trades.log` — no tracebacks.
3. **DB growth** — trade count should tick up on weekdays:
```bash
su - poteapp -c 'cd pote && source venv/bin/activate && python -c "
from sqlalchemy import func, select
from pote.db import SessionLocal
from pote.db.models import Trade, Official
with SessionLocal() as s:
print(\"trades\", s.scalar(select(func.count(Trade.id))))
print(\"officials\", s.scalar(select(func.count(Official.id))))
"'
```
---
**Vikunja:** [todo.levkin.ca → Business → POTE](https://todo.levkin.ca) (`POTE`)
## Open tasks (source of truth)
| P | Task | Owner | Status |
|---|------|-------|--------|
| **P1** | Proxmox backup schedule for LXC **236** on pve10 | @you | ⏳ |
| **P3** | Dedicated `pote@levkine.ca` mailbox (vs shared `alerts@`) | @you | optional / low |
| **P3** | Git deploy on LXC (replace rsync-only) | @agent | optional / low |
| **P3** | Kuma LAN health monitor | @agent | optional / low |
| **P3** | Mattermost / webhook alerts (email only today) | @agent | optional / low |
| **P3** | Full history ingest (second data source beyond kadoa cap) | @agent | optional / low |
**Closed 2026-07-11:** Beszel agent on pote-236 (`beszel-agent` active).
**Closed 2026-07-12:** `make deploy-pote` synced to LXC 236; 07:00 cron email delivered (`idobkin@gmail.com`, log OK). DB ~1129 trades / 80 officials.
### Not planned (unless you want them)
- Public URL / Caddy vhost (LAN-only by design)
- Investment signals exposed as advice (research descriptors only)
---
## Known issues / caveats
| Topic | Detail |
|-------|--------|
| **Disclosure lag** | STOCK Act filings appear weeks after trades; reports are descriptive, not timely trading signals. |
| **Amount ranges** | Disclosure buckets only ($1k$15k, etc.), not exact sizes. |
| **Empty tickers** | Some filings skipped when ticker missing. |
| **CI vs prod** | CI uses venv + Postgres service; prod uses host Postgres — both should pass after merge. |
| **Gitea deploy workflow** | `.github/workflows/deploy.yml` still references `git pull` on Proxmox; prod uses **rsync** via ansible `deploy-pote.sh`. |
---
## Related docs
| Doc | Purpose |
|-----|---------|
| [EMAIL_SETUP.md](EMAIL_SETUP.md) | SMTP / Mailcow / levkine.ca |
| [AUTOMATION_QUICKSTART.md](AUTOMATION_QUICKSTART.md) | Cron + reports |
| [PROXMOX_QUICKSTART.md](PROXMOX_QUICKSTART.md) | Original LXC provisioning |
| Ansible `docs/guides/projects-handoff-2026-05-26.md` | Multi-project homelab context |
| Ansible `docs/guides/smtp-inventory.md` | Mailboxes |
---
## One-line summary
**POTE on 10.0.10.48 ingests public congressional trades daily, emails a research summary to Gmail at 07:00, and is maintained via `main` + `make deploy-pote` — verify tomorrows cron email, then Beszel, backups, and optional data-source expansion.**
@@ -1,6 +1,6 @@
# Local Testing Guide for POTE # Local Testing Guide for POTE
## Testing Locally Before Deployment ## Testing Locally Before Deployment
### Quick Test - Run Full Suite ### Quick Test - Run Full Suite
@@ -10,18 +10,18 @@ source venv/bin/activate
pytest -v pytest -v
``` ```
**Expected Result:** All 55 tests should pass **Expected Result:** All 55 tests should pass
--- ---
## 📊 Current Data Status ## Current Data Status
### Live Data Status: **NOT LIVE YET** ### Live Data Status: **NOT LIVE YET**
**Why?** **Why?**
- 🔴 **House Stock Watcher API is DOWN** (domain issues, unreachable) - **House Stock Watcher API is DOWN** (domain issues, unreachable)
- 🟢 **yfinance works** (for price data) - **yfinance works** (for price data)
- 🟡 **Sample data available** (5 trades from fixtures) - **Sample data available** (5 trades from fixtures)
### What Data Do You Have? ### What Data Do You Have?
@@ -40,7 +40,7 @@ This will show:
--- ---
## 🧪 Testing Analytics Locally ## Testing Analytics Locally
### 1. Unit Tests (Fast, No External Dependencies) ### 1. Unit Tests (Fast, No External Dependencies)
@@ -53,10 +53,10 @@ pytest tests/test_analytics_integration.py -v
``` ```
These tests: These tests:
- Create synthetic price data - Create synthetic price data
- Simulate trades with known returns - Simulate trades with known returns
- Verify calculations are correct - Verify calculations are correct
- Test edge cases (missing data, sell trades, etc.) - Test edge cases (missing data, sell trades, etc.)
### 2. Manual Test with Local Database ### 2. Manual Test with Local Database
@@ -92,27 +92,27 @@ calc = ReturnCalculator(session)
--- ---
## 📦 What Gets Tested? ## What Gets Tested?
### Core Functionality (All Working ) ### Core Functionality (All Working )
1. **Database Models** - Officials, Securities, Trades, Prices 1. **Database Models** - Officials, Securities, Trades, Prices
2. **Data Ingestion** - Trade loading, security enrichment 2. **Data Ingestion** - Trade loading, security enrichment
3. **Analytics Engine** - Returns, benchmarks, metrics 3. **Analytics Engine** - Returns, benchmarks, metrics
4. **Edge Cases** - Missing data, sell trades, disclosure lags 4. **Edge Cases** - Missing data, sell trades, disclosure lags
### Integration Tests Cover: ### Integration Tests Cover:
- Return calculations over multiple time windows (30/60/90/180 days) - Return calculations over multiple time windows (30/60/90/180 days)
- Benchmark comparisons (stock vs SPY/QQQ) - Benchmark comparisons (stock vs SPY/QQQ)
- Abnormal return (alpha) calculations - Abnormal return (alpha) calculations
- Official performance summaries - Official performance summaries
- Sector analysis - Sector analysis
- Disclosure timing analysis - Disclosure timing analysis
- Top performer rankings - Top performer rankings
- System-wide statistics - System-wide statistics
--- ---
## 🔄 Getting Live Data ## Getting Live Data
### Option 1: Wait for House Stock Watcher API ### Option 1: Wait for House Stock Watcher API
The API is currently down. Once it's back up: The API is currently down. Once it's back up:
@@ -164,7 +164,7 @@ export QUIVER_API_KEY="your_key_here"
--- ---
## 📈 After Adding Data, Fetch Prices ## After Adding Data, Fetch Prices
```bash ```bash
# This will fetch prices for all securities in your database # This will fetch prices for all securities in your database
@@ -176,12 +176,12 @@ python scripts/enrich_securities.py
--- ---
## 🎯 Complete Local Test Workflow ## Complete Local Test Workflow
```bash ```bash
# 1. Run all tests # 1. Run all tests
pytest -v pytest -v
# All 55 tests should pass # All 55 tests should pass
# 2. Check local database # 2. Check local database
python -c " python -c "
@@ -210,7 +210,7 @@ python scripts/calculate_all_returns.py --window 90
--- ---
## 🚀 Deploy to Proxmox ## Deploy to Proxmox
Once local tests pass: Once local tests pass:
@@ -240,7 +240,7 @@ alembic upgrade head
--- ---
## 🐛 Common Issues ## Common Issues
### "No price data found" ### "No price data found"
**Fix:** Run `python scripts/fetch_sample_prices.py` **Fix:** Run `python scripts/fetch_sample_prices.py`
@@ -260,7 +260,7 @@ sudo -u postgres psql -c "\l"
--- ---
## 📊 Test Coverage ## Test Coverage
Run tests with coverage report: Run tests with coverage report:
@@ -277,19 +277,19 @@ firefox htmlcov/index.html # View coverage report
--- ---
## Summary ## Summary
**Before Deploying:** **Before Deploying:**
1. Run `pytest -v` - all tests pass 1. Run `pytest -v` - all tests pass
2. Run `make lint` - no errors 2. Run `make lint` - no errors
3. Test locally with sample data 3. Test locally with sample data
4. Verify analytics work with synthetic prices 4. Verify analytics work with synthetic prices
**Getting Live Data:** **Getting Live Data:**
- 🔴 House Stock Watcher API is down (external issue) - House Stock Watcher API is down (external issue)
- 🟢 Manual CSV import works NOW - Manual CSV import works NOW
- 🟢 yfinance for prices works NOW - yfinance for prices works NOW
- 🟡 QuiverQuant available (requires free API key) - QuiverQuant available (requires free API key)
**You can deploy and use the system NOW with:** **You can deploy and use the system NOW with:**
- Manual data entry - Manual data entry
+16 -16
View File
@@ -1,6 +1,6 @@
# Offline Demo - Works Without Internet! # Offline Demo - Works Without Internet!
## Full System Working Without Network Access ## Full System Working Without Network Access
Even though your environment doesn't have external internet, **everything works perfectly** using fixture files. Even though your environment doesn't have external internet, **everything works perfectly** using fixture files.
@@ -10,9 +10,9 @@ Even though your environment doesn't have external internet, **everything works
python scripts/ingest_from_fixtures.py python scripts/ingest_from_fixtures.py
# Output: # Output:
# Officials created/updated: 4 # Officials created/updated: 4
# Securities created/updated: 2 # Securities created/updated: 2
# Trades ingested: 5 # Trades ingested: 5
# #
# Database totals: # Database totals:
# Total officials: 4 # Total officials: 4
@@ -33,7 +33,7 @@ python scripts/ingest_from_fixtures.py
- NVDA, MSFT, AAPL, TSLA, GOOGL tickers - NVDA, MSFT, AAPL, TSLA, GOOGL tickers
2. **Offline Scripts** 2. **Offline Scripts**
- `scripts/ingest_from_fixtures.py` - Ingest sample trades ( works now!) - `scripts/ingest_from_fixtures.py` - Ingest sample trades ( works now!)
- `scripts/fetch_sample_prices.py` - Would need network (yfinance) - `scripts/fetch_sample_prices.py` - Would need network (yfinance)
3. **28 Passing Tests** - All use mocks, no network required 3. **28 Passing Tests** - All use mocks, no network required
@@ -67,14 +67,14 @@ with SessionLocal() as session:
### What You Can Do Offline ### What You Can Do Offline
**Run all tests**: `make test` **Run all tests**: `make test`
**Ingest fixture data**: `python scripts/ingest_from_fixtures.py` **Ingest fixture data**: `python scripts/ingest_from_fixtures.py`
**Query the database**: Use Python REPL or SQLite browser **Query the database**: Use Python REPL or SQLite browser
**Lint & format**: `make lint format` **Lint & format**: `make lint format`
**Run migrations**: `make migrate` **Run migrations**: `make migrate`
**Build analytics** (Phase 2): All math/ML works offline! **Build analytics** (Phase 2): All math/ML works offline!
**Can't do (needs network)**: **Can't do (needs network)**:
- Fetch live congressional trades from House Stock Watcher - Fetch live congressional trades from House Stock Watcher
- Fetch stock prices from yfinance - Fetch stock prices from yfinance
- (But you can add more fixture files to simulate this!) - (But you can add more fixture files to simulate this!)
@@ -108,9 +108,9 @@ You can expand the fixtures for offline development:
## Summary ## Summary
**The network error is not a problem!** The entire system is designed to work with: **The network error is not a problem!** The entire system is designed to work with:
- Fixtures for development/testing - Fixtures for development/testing
- Real APIs for production (when network available) - Real APIs for production (when network available)
- Same code paths for both - Same code paths for both
This is **by design** - makes development fast and tests reliable! 🚀 This is **by design** - makes development fast and tests reliable!
+6 -6
View File
@@ -223,12 +223,12 @@ print(f"Win Rate: {pelosi_stats['win_rate']:.1%}")
## Success Criteria ## Success Criteria
- Can calculate returns for any trade + window - Can calculate returns for any trade + window
- Can compare to S&P 500 benchmark - Can compare to S&P 500 benchmark
- Can generate official performance summaries - Can generate official performance summaries
- All calculations tested and accurate - All calculations tested and accurate
- Performance data stored efficiently - Performance data stored efficiently
- Documentation complete - Documentation complete
## Timeline ## Timeline
@@ -1,4 +1,4 @@
# Proxmox Quick Start # Proxmox Quick Start
**Got Proxmox? Deploy POTE in 5 minutes!** **Got Proxmox? Deploy POTE in 5 minutes!**
@@ -17,7 +17,7 @@ su - poteapp
cd pote && source venv/bin/activate cd pote && source venv/bin/activate
python scripts/ingest_from_fixtures.py python scripts/ingest_from_fixtures.py
# Done! # Done!
``` ```
--- ---
@@ -81,8 +81,8 @@ source venv/bin/activate
python scripts/ingest_from_fixtures.py python scripts/ingest_from_fixtures.py
# Should see: # Should see:
# Officials created: 4 # Officials created: 4
# Trades ingested: 5 # Trades ingested: 5
``` ```
### 5. Setup Cron Jobs ### 5. Setup Cron Jobs
@@ -96,7 +96,7 @@ crontab -e
15 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/enrich_securities.py >> /home/poteapp/logs/enrich.log 2>&1 15 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/enrich_securities.py >> /home/poteapp/logs/enrich.log 2>&1
``` ```
### 6. Done! 🎉 ### 6. Done!
Your POTE instance is now running and will: Your POTE instance is now running and will:
- Fetch congressional trades daily at 6 AM - Fetch congressional trades daily at 6 AM
@@ -107,12 +107,12 @@ Your POTE instance is now running and will:
## What You Get ## What You Get
**Full PostgreSQL database** **Full PostgreSQL database**
**Automated daily updates** (via cron) **Automated daily updates** (via cron)
**Isolated environment** (LXC container) **Isolated environment** (LXC container)
**Easy backups** (Proxmox snapshots) **Easy backups** (Proxmox snapshots)
**Low resource usage** (~500MB RAM) **Low resource usage** (~500MB RAM)
**Cost**: Just electricity (~$5-10/mo) **Cost**: Just electricity (~$5-10/mo)
--- ---
@@ -231,13 +231,13 @@ pip install -e .
## Next Steps ## Next Steps
1. Container running 1. Container running
2. POTE installed 2. POTE installed
3. Data ingested 3. Data ingested
4. ⏭ Setup Proxmox backups (Web UI → Datacenter → Backup) 4. ⏭ Setup Proxmox backups (Web UI → Datacenter → Backup)
5. ⏭ Configure static IP (if needed) 5. ⏭ Configure static IP (if needed)
6. ⏭ Build Phase 2 analytics 6. ⏭ Build Phase 2 analytics
7. ⏭ Add FastAPI dashboard 7. ⏭ Add FastAPI dashboard
--- ---
@@ -264,10 +264,10 @@ pct restart 100
--- ---
**Your Proxmox = Enterprise infrastructure at hobby prices!** 🚀 **Your Proxmox = Enterprise infrastructure at hobby prices!**
Cost breakdown: Cost breakdown:
- Cloud VPS: $20/mo - Cloud VPS: $20/mo
- Your Proxmox: ~$10/mo (power) - Your Proxmox: ~$10/mo (power)
- **Savings: $120/year** - **Savings: $120/year**
+19 -19
View File
@@ -1,15 +1,15 @@
# POTE Quick Start Guide # POTE Quick Start Guide
## 🚀 Your System is Ready! ## Your System is Ready!
**Container IP**: Check with `ip addr show eth0 | grep "inet"` **Container IP**: Check with `ip addr show eth0 | grep "inet"`
**Database**: PostgreSQL on port 5432 **Database**: PostgreSQL on port 5432
**Username**: `poteuser` **Username**: `poteuser`
**Password**: `changeme123` (⚠️ change in production!) **Password**: `changeme123` ( change in production!)
--- ---
## 📊 How to Use POTE ## How to Use POTE
### Option 1: Command Line (SSH into container) ### Option 1: Command Line (SSH into container)
@@ -51,7 +51,7 @@ with engine.connect() as conn:
--- ---
## 🎯 Common Tasks ## Common Tasks
### 1. Check System Status ### 1. Check System Status
@@ -145,7 +145,7 @@ ORDER BY trade_count DESC;
--- ---
## 📈 Example Workflows ## Example Workflows
### Workflow 1: Daily Update ### Workflow 1: Daily Update
@@ -237,7 +237,7 @@ print(f"Exported {len(df)} trades to trades_export.csv")
--- ---
## 🔧 Maintenance ## Maintenance
### Update POTE Code ### Update POTE Code
@@ -287,7 +287,7 @@ nano ~/pote/.env
--- ---
## 🌐 Access Methods Summary ## Access Methods Summary
| Method | From Where | Command | | Method | From Where | Command |
|--------|-----------|---------| |--------|-----------|---------|
@@ -298,22 +298,22 @@ nano ~/pote/.env
--- ---
## 📚 What Data Do You Have? ## What Data Do You Have?
Right now (Phase 1 complete): Right now (Phase 1 complete):
- **Congressional trading data** (from House Stock Watcher) - **Congressional trading data** (from House Stock Watcher)
- **Security information** (tickers, names, sectors) - **Security information** (tickers, names, sectors)
- **Historical prices** (OHLCV data from yfinance) - **Historical prices** (OHLCV data from yfinance)
- **Official profiles** (name, party, chamber, state) - **Official profiles** (name, party, chamber, state)
Coming next (Phase 2): Coming next (Phase 2):
- 📊 **Abnormal return calculations** - **Abnormal return calculations**
- 🤖 **Behavioral clustering** - **Behavioral clustering**
- 🚨 **Research signals** (follow_research, avoid_risk, watch) - **Research signals** (follow_research, avoid_risk, watch)
--- ---
## 🎓 Learning SQL for POTE ## Learning SQL for POTE
### Count Records ### Count Records
```sql ```sql
@@ -349,7 +349,7 @@ GROUP BY o.party;
--- ---
## Troubleshooting ## Troubleshooting
### Can't connect remotely? ### Can't connect remotely?
```bash ```bash
@@ -380,12 +380,12 @@ pip install -e .
--- ---
## 🚀 Next Steps ## Next Steps
1. **Populate with real data**: Run `fetch_congressional_trades.py` regularly 1. **Populate with real data**: Run `fetch_congressional_trades.py` regularly
2. **Set up cron job** for automatic daily updates 2. **Set up cron job** for automatic daily updates
3. **Build analytics** (Phase 2) - abnormal returns, signals 3. **Build analytics** (Phase 2) - abnormal returns, signals
4. **Create dashboard** (Phase 3) - web interface for exploration 4. **Create dashboard** (Phase 3) - web interface for exploration
Ready to build Phase 2 analytics? Just ask! 📈 Ready to build Phase 2 analytics? Just ask!
@@ -1,15 +1,15 @@
# 🚀 POTE Quick Setup Card # POTE Quick Setup Card
## 📍 Your Configuration ## Your Configuration
**Email Server:** `mail.levkin.ca` **Email Server:** `mail.levkin.ca`
**Email Account:** `test@levkin.ca` **Email Account:** `test@levkin.ca`
**Database:** PostgreSQL (configured) **Database:** PostgreSQL (configured)
**Status:** Ready for deployment **Status:** Ready for deployment
--- ---
## 3-Step Setup ## 3-Step Setup
### Step 1: Add Your Password (30 seconds) ### Step 1: Add Your Password (30 seconds)
@@ -27,7 +27,7 @@ source venv/bin/activate
python scripts/send_daily_report.py --to test@levkin.ca --test-smtp python scripts/send_daily_report.py --to test@levkin.ca --test-smtp
``` ```
**Check test@levkin.ca inbox** - you should receive a test email! **Check test@levkin.ca inbox** - you should receive a test email!
### Step 3: Automate (2 minutes) ### Step 3: Automate (2 minutes)
@@ -37,13 +37,13 @@ python scripts/send_daily_report.py --to test@levkin.ca --test-smtp
# Choose time: 6 AM (recommended) # Choose time: 6 AM (recommended)
``` ```
**Done!** 🎉 You'll now receive: **Done!** You'll now receive:
- Daily reports at 6 AM - Daily reports at 6 AM
- Weekly reports on Sundays - Weekly reports on Sundays
--- ---
## 📦 Deployment to Proxmox (5 minutes) ## Deployment to Proxmox (5 minutes)
### On Proxmox Host: ### On Proxmox Host:
@@ -79,7 +79,7 @@ bash scripts/proxmox_setup.sh
--- ---
## 🔍 Quick Commands ## Quick Commands
### On Deployed Server (SSH) ### On Deployed Server (SSH)
@@ -105,7 +105,7 @@ ls -lh ~/logs/*.txt
--- ---
## 📧 Email Configuration (.env) ## Email Configuration (.env)
```env ```env
SMTP_HOST=mail.levkin.ca SMTP_HOST=mail.levkin.ca
@@ -123,27 +123,27 @@ REPORT_RECIPIENTS=test@levkin.ca,user2@example.com,user3@example.com
--- ---
## 📊 What You'll Receive ## What You'll Receive
### Daily Report (6 AM) ### Daily Report (6 AM)
``` ```
New congressional trades New congressional trades
Market alerts (unusual activity) Market alerts (unusual activity)
Suspicious timing detections Suspicious timing detections
Summary statistics Summary statistics
``` ```
### Weekly Report (Sunday 8 AM) ### Weekly Report (Sunday 8 AM)
``` ```
Most active officials Most active officials
Most traded securities Most traded securities
Repeat offenders Repeat offenders
Pattern analysis Pattern analysis
``` ```
--- ---
## 🔧 Troubleshooting ## Troubleshooting
### Email Not Working? ### Email Not Working?
@@ -191,19 +191,19 @@ tail -50 ~/logs/daily_run.log
--- ---
## 📚 Documentation ## Documentation
| Document | Purpose | | Document | Purpose |
|----------|---------| |----------|---------|
| **[EMAIL_SETUP.md](EMAIL_SETUP.md)** | Your levkin.ca setup guide | | **[EMAIL_SETUP.md](EMAIL_SETUP.md)** | Your levkin.ca setup guide |
| **[DEPLOYMENT_AND_AUTOMATION.md](DEPLOYMENT_AND_AUTOMATION.md)** | Answers all questions | | **[DEPLOYMENT_AND_AUTOMATION.md](DEPLOYMENT_AND_AUTOMATION.md)** | Answers all questions |
| **[AUTOMATION_QUICKSTART.md](AUTOMATION_QUICKSTART.md)** | Quick automation guide | | **[AUTOMATION_QUICKSTART.md](AUTOMATION_QUICKSTART.md)** | Quick automation guide |
| **[PROXMOX_QUICKSTART.md](PROXMOX_QUICKSTART.md)** | Proxmox deployment | | **[PROXMOX_QUICKSTART.md](PROXMOX_QUICKSTART.md)** | Proxmox deployment |
| **[QUICKSTART.md](QUICKSTART.md)** | Usage guide | | **[QUICKSTART.md](QUICKSTART.md)** | Usage guide |
--- ---
## Checklist ## Checklist
**Local Development:** **Local Development:**
- [ ] `.env` file created with password - [ ] `.env` file created with password
@@ -227,18 +227,18 @@ tail -50 ~/logs/daily_run.log
--- ---
## 🎯 Your Current Status ## Your Current Status
**Code:** Complete (93 tests passing) **Code:** Complete (93 tests passing)
**Monitoring:** 3-phase system operational **Monitoring:** 3-phase system operational
**CI/CD:** Pipeline ready (.github/workflows/ci.yml) **CI/CD:** Pipeline ready (.github/workflows/ci.yml)
**Email:** Configured for test@levkin.ca **Email:** Configured for test@levkin.ca
**Deployment:** Ready to deploy to Proxmox **Deployment:** Ready to deploy to Proxmox
**Automation:** Ready to set up with `setup_cron.sh` **Automation:** Ready to set up with `setup_cron.sh`
--- ---
## 🚀 Next Action ## Next Action
**Right now (local testing):** **Right now (local testing):**
```bash ```bash
@@ -255,9 +255,9 @@ cd ~/pote
./scripts/setup_cron.sh ./scripts/setup_cron.sh
``` ```
**That's it! 🎉** **That's it! **
--- ---
**Everything is ready - just add your password and test!** 📧 **Everything is ready - just add your password and test!**
+27 -27
View File
@@ -3,25 +3,25 @@
**Last Updated**: 2025-12-14 **Last Updated**: 2025-12-14
**Version**: Phase 1 Complete (PR1 + PR2) **Version**: Phase 1 Complete (PR1 + PR2)
## 🎉 What's Working Now ## What's Working Now
### Data Ingestion (FREE!) ### Data Ingestion (FREE!)
**Congressional Trades**: Live ingestion from House Stock Watcher **Congressional Trades**: Live ingestion from House Stock Watcher
**Stock Prices**: Daily OHLCV from yfinance **Stock Prices**: Daily OHLCV from yfinance
**Officials**: Auto-populated from trade disclosures **Officials**: Auto-populated from trade disclosures
**Securities**: Auto-created, ready for enrichment **Securities**: Auto-created, ready for enrichment
### Database ### Database
**Schema**: Normalized (officials, securities, trades, prices, metrics stubs) **Schema**: Normalized (officials, securities, trades, prices, metrics stubs)
**Migrations**: Alembic configured and applied **Migrations**: Alembic configured and applied
**DB**: SQLite for dev, PostgreSQL-ready **DB**: SQLite for dev, PostgreSQL-ready
### Code Quality ### Code Quality
**Tests**: 28 passing (86% coverage) **Tests**: 28 passing (86% coverage)
**Linting**: ruff + mypy all green **Linting**: ruff + mypy all green
**Format**: black applied consistently **Format**: black applied consistently
## 📊 Current Stats ## Current Stats
```bash ```bash
# Test Suite # Test Suite
@@ -43,7 +43,7 @@ All free/open-source:
- pytest (testing) - pytest (testing)
``` ```
## 🚀 Quick Commands ## Quick Commands
### Fetch Live Data (FREE!) ### Fetch Live Data (FREE!)
```bash ```bash
@@ -75,7 +75,7 @@ make format # Format with black
make migrate # Run Alembic migrations make migrate # Run Alembic migrations
``` ```
## 🏠 Deployment ## Deployment
**Your Proxmox?** Perfect! See [`docs/08_proxmox_deployment.md`](docs/08_proxmox_deployment.md) for: **Your Proxmox?** Perfect! See [`docs/08_proxmox_deployment.md`](docs/08_proxmox_deployment.md) for:
- LXC container setup (lightweight, recommended) - LXC container setup (lightweight, recommended)
@@ -90,7 +90,7 @@ Other options in [`docs/07_deployment.md`](docs/07_deployment.md):
- Railway/Fly.io - $5-15/mo - Railway/Fly.io - $5-15/mo
- AWS/GCP - $20-50/mo - AWS/GCP - $20-50/mo
## 📂 Project Structure ## Project Structure
``` ```
pote/ pote/
@@ -138,7 +138,7 @@ pote/
└── fetch_sample_prices.py # Live price fetch └── fetch_sample_prices.py # Live price fetch
``` ```
## 💰 Cost Breakdown ## Cost Breakdown
| Component | Cost | Notes | | Component | Cost | Notes |
|-----------|------|-------| |-----------|------|-------|
@@ -154,7 +154,7 @@ Optional paid upgrades (NOT needed):
- Financial Modeling Prep: $15/mo (250 calls/day free tier available) - Financial Modeling Prep: $15/mo (250 calls/day free tier available)
- PostgreSQL hosting: $7+/mo (only if deploying) - PostgreSQL hosting: $7+/mo (only if deploying)
## Completed PRs ## Completed PRs
### PR1: Project Scaffold + Price Loader ### PR1: Project Scaffold + Price Loader
- [x] Project structure (`src/`, `tests/`, docs) - [x] Project structure (`src/`, `tests/`, docs)
@@ -176,7 +176,7 @@ Optional paid upgrades (NOT needed):
**See**: [`docs/PR2_SUMMARY.md`](docs/PR2_SUMMARY.md) **See**: [`docs/PR2_SUMMARY.md`](docs/PR2_SUMMARY.md)
## 📋 Next Steps (Phase 2 - Analytics) ## Next Steps (Phase 2 - Analytics)
### PR3: Security Enrichment ### PR3: Security Enrichment
- [ ] Enrich securities table with yfinance (names, sectors, exchanges) - [ ] Enrich securities table with yfinance (names, sectors, exchanges)
@@ -206,20 +206,20 @@ Optional paid upgrades (NOT needed):
**See**: [`docs/00_mvp.md`](docs/00_mvp.md) for full roadmap **See**: [`docs/00_mvp.md`](docs/00_mvp.md) for full roadmap
## 🔬 Research-Only Reminder ## Research-Only Reminder
**This tool is for private research and transparency analysis only.** **This tool is for private research and transparency analysis only.**
- Not investment advice - Not investment advice
- Not a trading system - Not a trading system
- No claims about inside information - No claims about inside information
- Public data only - Public data only
- Descriptive analytics - Descriptive analytics
- Research transparency - Research transparency
See [`docs/04_safety_ethics.md`](docs/04_safety_ethics.md) for guardrails. See [`docs/04_safety_ethics.md`](docs/04_safety_ethics.md) for guardrails.
## 🤝 Contributing ## Contributing
This is a personal research project, but if you want to use it: This is a personal research project, but if you want to use it:
@@ -229,7 +229,7 @@ This is a personal research project, but if you want to use it:
4. `python scripts/fetch_congressional_trades.py --days 7` 4. `python scripts/fetch_congressional_trades.py --days 7`
5. Start exploring! 5. Start exploring!
## 📄 License ## License
MIT License (for research/educational use only) MIT License (for research/educational use only)
+39 -39
View File
@@ -1,12 +1,12 @@
# POTE Testing Status Report # POTE Testing Status Report
**Date:** December 15, 2025 **Date:** December 15, 2025
**Status:** All Systems Operational - Ready for Deployment **Status:** All Systems Operational - Ready for Deployment
--- ---
## 🎯 Test Suite Summary ## Test Suite Summary
### **55 Tests - All Passing ** ### **55 Tests - All Passing **
``` ```
Platform: Python 3.13.5, pytest-9.0.2 Platform: Python 3.13.5, pytest-9.0.2
@@ -18,17 +18,17 @@ Coverage: ~85% overall
| Module | Tests | Status | Coverage | | Module | Tests | Status | Coverage |
|--------|-------|--------|----------| |--------|-------|--------|----------|
| **Analytics** | 18 tests | PASS | 80% | | **Analytics** | 18 tests | PASS | 80% |
| **Models** | 7 tests | PASS | 90% | | **Models** | 7 tests | PASS | 90% |
| **Ingestion** | 14 tests | PASS | 85% | | **Ingestion** | 14 tests | PASS | 85% |
| **Price Loader** | 8 tests | PASS | 90% | | **Price Loader** | 8 tests | PASS | 90% |
| **Security Enricher** | 8 tests | PASS | 85% | | **Security Enricher** | 8 tests | PASS | 85% |
--- ---
## 📊 What's Been Tested? ## What's Been Tested?
### Core Database Operations ### Core Database Operations
- [x] Creating and querying Officials - [x] Creating and querying Officials
- [x] Creating and querying Securities - [x] Creating and querying Securities
- [x] Creating and querying Trades - [x] Creating and querying Trades
@@ -36,7 +36,7 @@ Coverage: ~85% overall
- [x] Unique constraints and relationships - [x] Unique constraints and relationships
- [x] Database migrations (Alembic) - [x] Database migrations (Alembic)
### Data Ingestion ### Data Ingestion
- [x] House Stock Watcher client (with fixtures) - [x] House Stock Watcher client (with fixtures)
- [x] Trade loading from JSON - [x] Trade loading from JSON
- [x] Security enrichment from yfinance - [x] Security enrichment from yfinance
@@ -44,7 +44,7 @@ Coverage: ~85% overall
- [x] Idempotent operations (no duplicates) - [x] Idempotent operations (no duplicates)
- [x] Error handling for missing/invalid data - [x] Error handling for missing/invalid data
### Analytics Engine ### Analytics Engine
- [x] Return calculations (buy trades) - [x] Return calculations (buy trades)
- [x] Return calculations (sell trades) - [x] Return calculations (sell trades)
- [x] Multiple time windows (30/60/90/180 days) - [x] Multiple time windows (30/60/90/180 days)
@@ -56,7 +56,7 @@ Coverage: ~85% overall
- [x] Top performer rankings - [x] Top performer rankings
- [x] System-wide statistics - [x] System-wide statistics
### Edge Cases ### Edge Cases
- [x] Missing price data handling - [x] Missing price data handling
- [x] Trades with no exit price yet - [x] Trades with no exit price yet
- [x] Sell trades (inverted returns) - [x] Sell trades (inverted returns)
@@ -67,7 +67,7 @@ Coverage: ~85% overall
--- ---
## 🧪 Test Types ## Test Types
### 1. Unit Tests (Fast, Isolated) ### 1. Unit Tests (Fast, Isolated)
**Location:** `tests/test_*.py` (excluding integration) **Location:** `tests/test_*.py` (excluding integration)
@@ -100,7 +100,7 @@ Coverage: ~85% overall
--- ---
## 🔧 How to Run Tests Locally ## How to Run Tests Locally
### Quick Test ### Quick Test
```bash ```bash
@@ -136,7 +136,7 @@ ptw
--- ---
## 🚨 Known Limitations ## Known Limitations
### 1. External API Dependency ### 1. External API Dependency
**Issue:** House Stock Watcher API is currently DOWN **Issue:** House Stock Watcher API is currently DOWN
@@ -161,7 +161,7 @@ ptw
--- ---
## 📈 Performance Benchmarks ## Performance Benchmarks
### Test Execution Time ### Test Execution Time
- **Full suite:** 1.8 seconds - **Full suite:** 1.8 seconds
@@ -178,7 +178,7 @@ ptw
--- ---
## 🎯 Pre-Deployment Checklist ## Pre-Deployment Checklist
### Before Deploying to Proxmox: ### Before Deploying to Proxmox:
@@ -215,7 +215,7 @@ python scripts/enrich_securities.py
--- ---
## 🔄 Continuous Testing ## Continuous Testing
### Git Pre-Commit Hook (Optional) ### Git Pre-Commit Hook (Optional)
```bash ```bash
@@ -247,7 +247,7 @@ jobs:
--- ---
## 📝 Test Maintenance ## Test Maintenance
### Adding New Tests ### Adding New Tests
@@ -281,36 +281,36 @@ Fixtures are in `tests/conftest.py`:
--- ---
## 🎉 Summary ## Summary
### Current Status: **PRODUCTION READY** ### Current Status: **PRODUCTION READY**
**What Works:** **What Works:**
- All 55 tests passing - All 55 tests passing
- Full analytics pipeline functional - Full analytics pipeline functional
- Database operations solid - Database operations solid
- Data ingestion from multiple sources - Data ingestion from multiple sources
- Price fetching from yfinance - Price fetching from yfinance
- Security enrichment - Security enrichment
- Return calculations - Return calculations
- Benchmark comparisons - Benchmark comparisons
- Performance metrics - Performance metrics
- CLI scripts operational - CLI scripts operational
**What's Missing:** **What's Missing:**
- Live congressional trade API (external issue - House Stock Watcher down) - Live congressional trade API (external issue - House Stock Watcher down)
- **Workaround:** Manual import, CSV, or alternative APIs available - **Workaround:** Manual import, CSV, or alternative APIs available
**Next Steps:** **Next Steps:**
1. Tests are complete 1. Tests are complete
2. Code is ready 2. Code is ready
3. ➡️ **Deploy to Proxmox** (or continue with Phase 2 features) 3. **Deploy to Proxmox** (or continue with Phase 2 features)
4. ➡️ Add more data sources 4. Add more data sources
5. ➡️ Build dashboard (Phase 3) 5. Build dashboard (Phase 3)
--- ---
## 📞 Need Help? ## Need Help?
See: See:
- `LOCAL_TEST_GUIDE.md` - Detailed local testing instructions - `LOCAL_TEST_GUIDE.md` - Detailed local testing instructions
+18 -18
View File
@@ -1,6 +1,6 @@
# POTE Watchlist & Trading Reports # POTE Watchlist & Trading Reports
## 🎯 Get Trading Reports 1 Hour Before Market Close ## Get Trading Reports 1 Hour Before Market Close
### Quick Setup ### Quick Setup
@@ -25,7 +25,7 @@ crontab -e
--- ---
## 📋 Watchlist System ## Watchlist System
### Who's on the Default Watchlist? ### Who's on the Default Watchlist?
@@ -48,7 +48,7 @@ crontab -e
--- ---
## 🔧 Managing Your Watchlist ## Managing Your Watchlist
### View Current Watchlist ### View Current Watchlist
@@ -104,7 +104,7 @@ This fetches all 535 members of Congress (100 Senate + 435 House).
--- ---
## 📊 Generating Reports ## Generating Reports
### Manual Report Generation ### Manual Report Generation
@@ -135,24 +135,24 @@ python scripts/generate_trading_report.py --format json --output report.json
================================================================================ ================================================================================
──────────────────────────────────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────────────
👤 Nancy Pelosi (D-CA, House) Nancy Pelosi (D-CA, House)
──────────────────────────────────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────────────
Side Ticker Company Sector Value Trade Date Filed Side Ticker Company Sector Value Trade Date Filed
-------- ------ -------------------------- ---------- ------------------- ---------- ---------- -------- ------ -------------------------- ---------- ------------------- ---------- ----------
🟢 BUY NVDA NVIDIA Corporation Technology $15,001 - $50,000 2024-11-15 2024-12-01 BUY NVDA NVIDIA Corporation Technology $15,001 - $50,000 2024-11-15 2024-12-01
🔴 SELL MSFT Microsoft Corporation Technology $50,001 - $100,000 2024-11-20 2024-12-01 SELL MSFT Microsoft Corporation Technology $50,001 - $100,000 2024-11-20 2024-12-01
──────────────────────────────────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────────────
👤 Tommy Tuberville (R-AL, Senate) Tommy Tuberville (R-AL, Senate)
──────────────────────────────────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────────────
Side Ticker Company Sector Value Trade Date Filed Side Ticker Company Sector Value Trade Date Filed
-------- ------ -------------------------- ---------- ------------------- ---------- ---------- -------- ------ -------------------------- ---------- ------------------- ---------- ----------
🟢 BUY SPY SPDR S&P 500 ETF Financial $100,001 - $250,000 2024-11-18 2024-12-02 BUY SPY SPDR S&P 500 ETF Financial $100,001 - $250,000 2024-11-18 2024-12-02
🟢 BUY AAPL Apple Inc. Technology $50,001 - $100,000 2024-11-22 2024-12-02 BUY AAPL Apple Inc. Technology $50,001 - $100,000 2024-11-22 2024-12-02
🔴 SELL TSLA Tesla, Inc. Automotive $15,001 - $50,000 2024-11-25 2024-12-02 SELL TSLA Tesla, Inc. Automotive $15,001 - $50,000 2024-11-25 2024-12-02
================================================================================ ================================================================================
📊 SUMMARY SUMMARY
================================================================================ ================================================================================
Total Trades: 5 Total Trades: 5
@@ -223,7 +223,7 @@ Runs at 8 AM and 3 PM daily (weekdays).
--- ---
## 📧 Email Reports (Optional) ## Email Reports (Optional)
### Setup Email Notifications ### Setup Email Notifications
@@ -256,7 +256,7 @@ python scripts/send_email.py /tmp/report.html your-email@example.com
--- ---
## 🎯 Typical Workflow ## Typical Workflow
### Daily Routine (3 PM ET) ### Daily Routine (3 PM ET)
@@ -282,7 +282,7 @@ python scripts/send_email.py /tmp/report.html your-email@example.com
--- ---
## 🔍 Finding More Officials ## Finding More Officials
### Public Resources ### Public Resources
@@ -318,7 +318,7 @@ Add committee members to your watchlist.
--- ---
## 📈 Example Cron Setup ## Example Cron Setup
```bash ```bash
# Edit crontab # Edit crontab
@@ -341,7 +341,7 @@ This gives you:
--- ---
## 🚀 Quick Start Summary ## Quick Start Summary
```bash ```bash
# 1. Create watchlist # 1. Create watchlist
@@ -363,7 +363,7 @@ cat reports/trading_report_$(date +%Y%m%d).txt
--- ---
## FAQ ## FAQ
**Q: Why are all the trades old (30-45 days)?** **Q: Why are all the trades old (30-45 days)?**
**A:** Federal law (STOCK Act) gives Congress 30-45 days to file. This is normal. **A:** Federal law (STOCK Act) gives Congress 30-45 days to file. This is normal.
@@ -1,8 +1,8 @@
# 🎉 POTE Monitoring System - ALL PHASES COMPLETE! # POTE Monitoring System - ALL PHASES COMPLETE!
## **What Was Built (3 Phases)** ## **What Was Built (3 Phases)**
### **Phase 1: Real-Time Market Monitoring** ### **Phase 1: Real-Time Market Monitoring**
**Detects unusual market activity in congressional tickers** **Detects unusual market activity in congressional tickers**
**Features:** **Features:**
@@ -20,11 +20,11 @@
- `MarketAlert` model - Database storage - `MarketAlert` model - Database storage
- `monitor_market.py` - CLI tool - `monitor_market.py` - CLI tool
**Tests:** 14 passing **Tests:** 14 passing
--- ---
### **Phase 2: Disclosure Timing Correlation** ### **Phase 2: Disclosure Timing Correlation**
**Matches trades to prior market alerts when disclosures appear** **Matches trades to prior market alerts when disclosures appear**
**Features:** **Features:**
@@ -50,11 +50,11 @@
- `DisclosureCorrelator` - Correlation engine - `DisclosureCorrelator` - Correlation engine
- `analyze_disclosure_timing.py` - CLI tool - `analyze_disclosure_timing.py` - CLI tool
**Tests:** 13 passing **Tests:** 13 passing
--- ---
### **Phase 3: Pattern Detection & Rankings** ### **Phase 3: Pattern Detection & Rankings**
**Cross-official analysis and comparative rankings** **Cross-official analysis and comparative rankings**
**Features:** **Features:**
@@ -71,19 +71,19 @@
- `PatternDetector` - Pattern analysis engine - `PatternDetector` - Pattern analysis engine
- `generate_pattern_report.py` - CLI tool - `generate_pattern_report.py` - CLI tool
**Tests:** 11 passing **Tests:** 11 passing
--- ---
## 📊 **Complete System Architecture** ## **Complete System Architecture**
``` ```
┌─────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────┐
│ PHASE 1: Real-Time Monitoring │ │ PHASE 1: Real-Time Monitoring │
│ ──────────────────────────────────── │ │ ──────────────────────────────────── │
🔔 Monitor congressional tickers │ Monitor congressional tickers │
📊 Detect unusual activity │ Detect unusual activity │
💾 Log alerts to database │ Log alerts to database │
└─────────────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────────────┘
[30-45 days pass] [30-45 days pass]
@@ -91,25 +91,25 @@
┌─────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────┐
│ PHASE 2: Disclosure Correlation │ │ PHASE 2: Disclosure Correlation │
│ ─────────────────────────────── │ │ ─────────────────────────────── │
📋 New congressional trades filed │ New congressional trades filed │
🔗 Match to prior alerts │ Match to prior alerts │
📈 Calculate timing scores │ Calculate timing scores │
🚩 Flag suspicious trades │ Flag suspicious trades │
└─────────────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐ ┌─────────────────────────────────────────────────────────────┐
│ PHASE 3: Pattern Detection │ │ PHASE 3: Pattern Detection │
│ ────────────────────────── │ │ ────────────────────────── │
📊 Rank officials by timing │ Rank officials by timing │
🔥 Identify repeat offenders │ Identify repeat offenders │
📈 Compare parties, sectors, tickers │ Compare parties, sectors, tickers │
📋 Generate comprehensive reports │ Generate comprehensive reports │
└─────────────────────────────────────────────────────────────┘ └─────────────────────────────────────────────────────────────┘
``` ```
--- ---
## 🚀 **Usage Guide** ## **Usage Guide**
### **1. Set Up Monitoring (Run Daily)** ### **1. Set Up Monitoring (Run Daily)**
@@ -169,7 +169,7 @@ python scripts/generate_pattern_report.py --days 365 --format json --output patt
--- ---
## 📋 **Example Reports** ## **Example Reports**
### **Timing Analysis Report** ### **Timing Analysis Report**
@@ -179,7 +179,7 @@ python scripts/generate_pattern_report.py --days 365 --format json --output patt
3 Trades with Timing Advantages Detected 3 Trades with Timing Advantages Detected
================================================================================ ================================================================================
🚨 #1 - HIGHLY SUSPICIOUS (Timing Score: 85/100) #1 - HIGHLY SUSPICIOUS (Timing Score: 85/100)
──────────────────────────────────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────────────
Official: Nancy Pelosi Official: Nancy Pelosi
Ticker: NVDA Ticker: NVDA
@@ -187,16 +187,16 @@ Side: BUY
Trade Date: 2024-01-15 Trade Date: 2024-01-15
Value: $15,001-$50,000 Value: $15,001-$50,000
📊 Timing Analysis: Timing Analysis:
Prior Alerts: 3 Prior Alerts: 3
Recent Alerts (7d): 2 Recent Alerts (7d): 2
High Severity: 2 High Severity: 2
Avg Severity: 7.5/10 Avg Severity: 7.5/10
💡 Assessment: Trade occurred after 3 alerts, including 2 high-severity. Assessment: Trade occurred after 3 alerts, including 2 high-severity.
High likelihood of timing advantage. High likelihood of timing advantage.
🔔 Prior Market Alerts: Prior Market Alerts:
Timestamp Type Severity Timing Timestamp Type Severity Timing
2024-01-12 10:30:00 Unusual Volume 8/10 3 days before 2024-01-12 10:30:00 Unusual Volume 8/10 3 days before
2024-01-13 14:15:00 Price Spike 7/10 2 days before 2024-01-13 14:15:00 Price Spike 7/10 2 days before
@@ -211,27 +211,27 @@ Timestamp Type Severity Timing
Period: 365 days Period: 365 days
================================================================================ ================================================================================
📊 SUMMARY SUMMARY
──────────────────────────────────────────────────────────────────────────────── ────────────────────────────────────────────────────────────────────────────────
Officials Analyzed: 45 Officials Analyzed: 45
Repeat Offenders: 8 Repeat Offenders: 8
Average Timing Score: 42.3/100 Average Timing Score: 42.3/100
🚨 TOP 10 MOST SUSPICIOUS OFFICIALS (By Timing Score) TOP 10 MOST SUSPICIOUS OFFICIALS (By Timing Score)
================================================================================ ================================================================================
Rank Official Party-State Chamber Trades Suspicious Rate Avg Score Rank Official Party-State Chamber Trades Suspicious Rate Avg Score
──── ─────────────────────── ─────────── ─────── ────── ────────── ────── ───────── ──── ─────────────────────── ─────────── ─────── ────── ────────── ────── ─────────
🚨 1 Tommy Tuberville R-AL Senate 47 35/47 74.5% 72.5/100 1 Tommy Tuberville R-AL Senate 47 35/47 74.5% 72.5/100
🚨 2 Nancy Pelosi D-CA House 38 28/38 73.7% 71.2/100 2 Nancy Pelosi D-CA House 38 28/38 73.7% 71.2/100
🔴 3 Dan Crenshaw R-TX House 25 15/25 60.0% 65.8/100 3 Dan Crenshaw R-TX House 25 15/25 60.0% 65.8/100
🔴 4 Marjorie Taylor Greene R-GA House 19 11/19 57.9% 63.2/100 4 Marjorie Taylor Greene R-GA House 19 11/19 57.9% 63.2/100
🟡 5 Josh Gottheimer D-NJ House 31 14/31 45.2% 58.7/100 5 Josh Gottheimer D-NJ House 31 14/31 45.2% 58.7/100
🔥 REPEAT OFFENDERS (50%+ Suspicious Trades) REPEAT OFFENDERS (50%+ Suspicious Trades)
================================================================================ ================================================================================
🚨 Tommy Tuberville (R-AL, Senate) Tommy Tuberville (R-AL, Senate)
Trades: 47 | Suspicious: 35 (74.5%) Trades: 47 | Suspicious: 35 (74.5%)
Avg Timing Score: 72.5/100 Avg Timing Score: 72.5/100
Pattern: HIGHLY SUSPICIOUS - Majority of trades show timing advantage Pattern: HIGHLY SUSPICIOUS - Majority of trades show timing advantage
@@ -239,9 +239,9 @@ Rank Official Party-State Chamber Trades Suspicious Rate
--- ---
## 📈 **Test Coverage** ## **Test Coverage**
**Total: 93 tests, all passing ** **Total: 93 tests, all passing **
- **Phase 1 (Monitoring):** 14 tests - **Phase 1 (Monitoring):** 14 tests
- **Phase 2 (Correlation):** 13 tests - **Phase 2 (Correlation):** 13 tests
@@ -252,7 +252,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
--- ---
## 🎯 **Key Insights the System Provides** ## **Key Insights the System Provides**
### **1. Individual Official Analysis** ### **1. Individual Official Analysis**
- Which officials consistently trade before unusual activity? - Which officials consistently trade before unusual activity?
@@ -282,7 +282,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
--- ---
## 🔧 **Automated Workflow** ## **Automated Workflow**
### **Daily Routine (Recommended)** ### **Daily Routine (Recommended)**
@@ -299,7 +299,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
--- ---
## 📊 **Database Schema** ## **Database Schema**
**New Table: `market_alerts`** **New Table: `market_alerts`**
```sql ```sql
@@ -315,7 +315,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
--- ---
## 🎓 **Interpretation Guide** ## **Interpretation Guide**
### **Timing Scores** ### **Timing Scores**
- **80-100:** Highly suspicious - Multiple high-severity alerts before trade - **80-100:** Highly suspicious - Multiple high-severity alerts before trade
@@ -337,15 +337,15 @@ Rank Official Party-State Chamber Trades Suspicious Rate
--- ---
## ⚠️ **Important Disclaimers** ## **Important Disclaimers**
### **Legal & Ethical** ### **Legal & Ethical**
1. All data is public and legally obtained 1. All data is public and legally obtained
2. Analysis is retrospective (30-45 day lag) 2. Analysis is retrospective (30-45 day lag)
3. For research and transparency only 3. For research and transparency only
4. NOT investment advice 4. NOT investment advice
5. NOT proof of illegal activity (requires investigation) 5. NOT proof of illegal activity (requires investigation)
6. Statistical patterns ≠ legal evidence 6. Statistical patterns ≠ legal evidence
### **Technical Limitations** ### **Technical Limitations**
1. Cannot identify WHO is trading in real-time 1. Cannot identify WHO is trading in real-time
@@ -356,7 +356,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
--- ---
## 🚀 **Deployment Checklist** ## **Deployment Checklist**
### **On Proxmox Container** ### **On Proxmox Container**
@@ -383,7 +383,7 @@ python scripts/generate_pattern_report.py --days 365
--- ---
## 📚 **Documentation** ## **Documentation**
- **`docs/11_live_market_monitoring.md`** - Deep dive into monitoring - **`docs/11_live_market_monitoring.md`** - Deep dive into monitoring
- **`LOCAL_TEST_GUIDE.md`** - Testing instructions - **`LOCAL_TEST_GUIDE.md`** - Testing instructions
@@ -392,23 +392,23 @@ python scripts/generate_pattern_report.py --days 365
--- ---
## 🎉 **Achievement Unlocked!** ## **Achievement Unlocked!**
**You now have a complete system that:** **You now have a complete system that:**
Monitors real-time market activity Monitors real-time market activity
Correlates trades to prior alerts Correlates trades to prior alerts
Calculates timing advantage scores Calculates timing advantage scores
Identifies repeat offenders Identifies repeat offenders
Ranks officials by suspicion Ranks officials by suspicion
Generates comprehensive reports Generates comprehensive reports
93 tests confirming it works 93 tests confirming it works
**This is a production-ready transparency and research tool!** 🚀 **This is a production-ready transparency and research tool!**
--- ---
## 🔜 **Potential Future Enhancements** ## **Potential Future Enhancements**
### **Phase 4 Ideas (Optional)** ### **Phase 4 Ideas (Optional)**
- Email/SMS alerts for high-severity patterns - Email/SMS alerts for high-severity patterns
@@ -420,6 +420,6 @@ python scripts/generate_pattern_report.py --days 365
- Automated PDF reports - Automated PDF reports
- Historical performance tracking - Historical performance tracking
**But the core system is COMPLETE and FUNCTIONAL now!** **But the core system is COMPLETE and FUNCTIONAL now!**
@@ -1,6 +1,6 @@
# PR1 Summary: Project Scaffold + DB + Price Loader # PR1 Summary: Project Scaffold + DB + Price Loader
**Status**: Complete **Status**: Complete
**Date**: 2025-12-13 **Date**: 2025-12-13
## What was built ## What was built
@@ -36,7 +36,7 @@ Includes proper indexes, unique constraints, and relationships.
- `tests/conftest.py`: fixtures for in-memory DB, sample officials/securities/trades/prices - `tests/conftest.py`: fixtures for in-memory DB, sample officials/securities/trades/prices
- `tests/test_models.py`: model creation, relationships, unique constraints, queries (7 tests) - `tests/test_models.py`: model creation, relationships, unique constraints, queries (7 tests)
- `tests/test_price_loader.py`: loader logic, idempotency, upsert, mocking yfinance (8 tests) - `tests/test_price_loader.py`: loader logic, idempotency, upsert, mocking yfinance (8 tests)
- **Result**: 15 tests, all passing - **Result**: 15 tests, all passing
### 6. Tooling ### 6. Tooling
- **Black** + **ruff** configured and run (all code formatted + linted) - **Black** + **ruff** configured and run (all code formatted + linted)
@@ -1,6 +1,6 @@
# PR2 Summary: Congressional Trade Ingestion # PR2 Summary: Congressional Trade Ingestion
**Status**: Complete **Status**: Complete
**Date**: 2025-12-14 **Date**: 2025-12-14
## What was built ## What was built
@@ -26,7 +26,7 @@
- `tests/fixtures/sample_house_watcher.json`: 5 realistic sample transactions - `tests/fixtures/sample_house_watcher.json`: 5 realistic sample transactions
- Includes House + Senate, Democrats + Republicans, various tickers - Includes House + Senate, Democrats + Republicans, various tickers
### 4. Tests (13 new tests, all passing ) ### 4. Tests (13 new tests, all passing )
**`tests/test_house_watcher.py` (8 tests)**: **`tests/test_house_watcher.py` (8 tests)**:
- Amount range parsing (with range, single value, invalid) - Amount range parsing (with range, single value, invalid)
- Transaction type normalization - Transaction type normalization
@@ -58,9 +58,9 @@
python scripts/fetch_congressional_trades.py --days 30 python scripts/fetch_congressional_trades.py --days 30
# Sample output: # Sample output:
# Officials created/updated: 47 # Officials created/updated: 47
# Securities created/updated: 89 # Securities created/updated: 89
# Trades ingested: 234 # Trades ingested: 234
``` ```
### Database Queries ### Database Queries
@@ -1,6 +1,6 @@
# PR3 Summary: Security Enrichment + Deployment # PR3 Summary: Security Enrichment + Deployment
**Status**: Complete **Status**: Complete
**Date**: 2025-12-14 **Date**: 2025-12-14
## What was built ## What was built
@@ -32,7 +32,7 @@
python scripts/enrich_securities.py --force python scripts/enrich_securities.py --force
``` ```
### 3. Tests (9 new tests, all passing ) ### 3. Tests (9 new tests, all passing )
**`tests/test_security_enricher.py`**: **`tests/test_security_enricher.py`**:
- Successful enrichment with complete data - Successful enrichment with complete data
- ETF detection and classification - ETF detection and classification
@@ -72,7 +72,7 @@ python scripts/enrich_securities.py
# Enriched AAPL: Apple Inc. (Technology) # Enriched AAPL: Apple Inc. (Technology)
# Enriched TSLA: Tesla, Inc. (Consumer Cyclical) # Enriched TSLA: Tesla, Inc. (Consumer Cyclical)
# Enriched GOOGL: Alphabet Inc. (Communication Services) # Enriched GOOGL: Alphabet Inc. (Communication Services)
# Successfully enriched: 5 # Successfully enriched: 5
``` ```
### Query Enriched Data ### Query Enriched Data
@@ -159,10 +159,10 @@ python scripts/update_all_prices.py # To be built in PR4
| Option | Complexity | Cost/month | Best For | | Option | Complexity | Cost/month | Best For |
|--------|-----------|------------|----------| |--------|-----------|------------|----------|
| **Local** | | $0 | Development | | **Local** | | $0 | Development |
| **VPS + Docker** | ⭐⭐ | $10-20 | Personal deployment | | **VPS + Docker** | | $10-20 | Personal deployment |
| **Railway/Fly.io** | | $5-15 | Easy cloud | | **Railway/Fly.io** | | $5-15 | Easy cloud |
| **AWS** | ⭐⭐⭐ | $20-50 | Scalable production | | **AWS** | | $20-50 | Scalable production |
See [`docs/07_deployment.md`](07_deployment.md) for detailed guides. See [`docs/07_deployment.md`](07_deployment.md) for detailed guides.
@@ -1,6 +1,6 @@
# PR4 Summary: Phase 2 Analytics Foundation # PR4 Summary: Phase 2 Analytics Foundation
## Completed ## Completed
**Date**: December 15, 2025 **Date**: December 15, 2025
**Status**: Complete **Status**: Complete
@@ -78,14 +78,14 @@ python scripts/calculate_all_returns.py --window 90 --benchmark SPY --top 10
### 3. Tests (`tests/test_analytics.py`) ### 3. Tests (`tests/test_analytics.py`)
- Return calculator with sample data - Return calculator with sample data
- Buy vs sell trade handling - Buy vs sell trade handling
- Missing data edge cases - Missing data edge cases
- Benchmark comparisons - Benchmark comparisons
- Official performance metrics - Official performance metrics
- Multiple time windows - Multiple time windows
- Sector analysis - Sector analysis
- Timing analysis - Timing analysis
**Test Coverage**: Analytics module fully tested **Test Coverage**: Analytics module fully tested
@@ -288,15 +288,15 @@ with next(get_session()) as session:
- Trades near policy events - Trades near policy events
- Unusual timing flags - Unusual timing flags
## Success Criteria ## Success Criteria
- Can calculate returns for any trade + window - Can calculate returns for any trade + window
- Can compare to S&P 500 benchmark - Can compare to S&P 500 benchmark
- Can generate official performance summaries - Can generate official performance summaries
- All calculations tested and accurate - All calculations tested and accurate
- Performance data calculated on-the-fly - Performance data calculated on-the-fly
- Documentation complete - Documentation complete
- Command-line tools working - Command-line tools working
## Testing ## Testing
@@ -309,7 +309,7 @@ All analytics tests should pass (may have warnings if no price data).
--- ---
**Phase 2 Analytics Foundation: COMPLETE** **Phase 2 Analytics Foundation: COMPLETE**
**Ready for**: PR5 (Signals), PR6 (API), PR7 (Dashboard) **Ready for**: PR5 (Signals), PR6 (API), PR7 (Dashboard)
+3
View File
@@ -0,0 +1,3 @@
# Archive
PR summaries and one-shot status writeups. Prefer the numbered guides and QUICKSTART files.
+2
View File
@@ -41,6 +41,8 @@ where = ["src"]
[tool.ruff] [tool.ruff]
line-length = 100 line-length = 100
target-version = "py311" target-version = "py311"
[tool.ruff.lint]
select = ["E", "F", "W", "I", "N", "UP", "B", "A", "C4", "SIM", "RET"] select = ["E", "F", "W", "I", "N", "UP", "B", "A", "C4", "SIM", "RET"]
ignore = ["E501"] # Line too long (handled by black) ignore = ["E501"] # Line too long (handled by black)
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
# One-time installer for the local pre-commit gitleaks hook.
# Run once per clone: bash scripts/git-hooks/install.sh
#
# Respects `core.hooksPath` if you've set one (local or global) — some setups
# point git at a hooks dir outside `.git/hooks/` (e.g. a machine-wide
# `~/.git-hooks/`), and installing to `.git/hooks/` in that case would be a
# silent no-op. If an existing hook is already at that path, this chains to
# it so nothing already relying on it breaks.
set -euo pipefail
REPO_ROOT="$(git rev-parse --show-toplevel)"
cd "$REPO_ROOT"
HOOKS_DIR="$(git config --get core.hooksPath || true)"
if [ -z "$HOOKS_DIR" ]; then
HOOKS_DIR=".git/hooks"
elif [[ "$HOOKS_DIR" != /* ]]; then
HOOKS_DIR="$REPO_ROOT/$HOOKS_DIR"
fi
mkdir -p "$HOOKS_DIR"
TARGET="$HOOKS_DIR/pre-commit"
if [ -f "$TARGET" ] && ! grep -q "gitleaks" "$TARGET" 2>/dev/null; then
echo "⚠️ Existing pre-commit hook found at $TARGET that isn't ours — chaining instead of overwriting."
CHAINED="$HOOKS_DIR/pre-commit.d-gitleaks"
cp scripts/git-hooks/pre-commit "$CHAINED"
chmod +x "$CHAINED"
if ! grep -q "pre-commit.d-gitleaks" "$TARGET" 2>/dev/null; then
printf '\n# Added by levkinops ansible repo (scripts/git-hooks/install.sh)\n"%s"\n' "$CHAINED" >> "$TARGET"
fi
else
cp scripts/git-hooks/pre-commit "$TARGET"
chmod +x "$TARGET"
fi
echo "✓ Installed pre-commit gitleaks hook → $TARGET"
if [ "$HOOKS_DIR" != ".git/hooks" ] && [ "$HOOKS_DIR" != "$REPO_ROOT/.git/hooks" ]; then
echo " (using core.hooksPath=$HOOKS_DIR — applies to every repo that shares this hooksPath)"
fi
if ! command -v gitleaks >/dev/null 2>&1; then
echo " Note: gitleaks isn't installed locally yet. Install it for the hook to actually run:"
echo " macOS: brew install gitleaks"
echo " Linux: see https://github.com/gitleaks/gitleaks#installing"
echo " Until then this hook is a no-op locally (CI still scans every push)."
fi
+27
View File
@@ -0,0 +1,27 @@
#!/usr/bin/env bash
# Local pre-commit secret scan — mirrors the `secret-scanning` CI job (gitleaks)
# so leaked secrets are caught before they ever leave your machine, not just at
# CI time. Installed via `make install-git-hooks`.
#
# Uses the same .gitleaks.toml allowlist as CI. Only scans staged content.
set -euo pipefail
REPO_ROOT="$(git rev-parse --show-toplevel)"
cd "$REPO_ROOT"
if ! command -v gitleaks >/dev/null 2>&1; then
echo "⚠️ gitleaks not installed locally — skipping local secret scan."
echo " Install: brew install gitleaks (CI will still catch it either way)"
exit 0
fi
echo "🔐 Running gitleaks on staged changes..."
if ! gitleaks protect --staged --config .gitleaks.toml --no-banner --redact; then
echo ""
echo "❌ gitleaks found a potential secret in your staged changes."
echo " Fix it, or if it's a false positive, add an allowlist entry to .gitleaks.toml."
echo " Bypass (not recommended): git commit --no-verify"
exit 1
fi
echo "✓ gitleaks: no secrets found in staged changes"
+1 -1
View File
@@ -74,7 +74,7 @@ def main(tickers, interval, once, min_severity, save_report, lookback):
if filtered: if filtered:
# Generate report # Generate report
report = alert_mgr.generate_summary_report(filtered, format="text") report = alert_mgr.generate_summary_report(filtered, output_format="text")
print("\n" + report) print("\n" + report)
# Save report if requested # Save report if requested
+9 -1
View File
@@ -43,6 +43,12 @@ def main():
action="store_true", action="store_true",
help="Test SMTP connection before sending", help="Test SMTP connection before sending",
) )
parser.add_argument(
"--lookback-days",
type=int,
default=1,
help="Include trades filed in the last N days ending on the report date (default: 1)",
)
parser.add_argument( parser.add_argument(
"--save-to-file", "--save-to-file",
help="Also save report to this file path", help="Also save report to this file path",
@@ -81,7 +87,9 @@ def main():
logger.info(f"Generating daily report for {report_date or date.today()}...") logger.info(f"Generating daily report for {report_date or date.today()}...")
with get_session() as session: with get_session() as session:
generator = ReportGenerator(session) generator = ReportGenerator(session)
report_data = generator.generate_daily_summary(report_date) report_data = generator.generate_daily_summary(
report_date, lookback_days=args.lookback_days
)
# Format as text and HTML # Format as text and HTML
text_body = generator.format_as_text(report_data, "daily") text_body = generator.format_as_text(report_data, "daily")
+1 -3
View File
@@ -2,14 +2,12 @@
Analytics module for calculating returns, performance metrics, and signals. Analytics module for calculating returns, performance metrics, and signals.
""" """
from .returns import ReturnCalculator
from .benchmarks import BenchmarkComparison from .benchmarks import BenchmarkComparison
from .metrics import PerformanceMetrics from .metrics import PerformanceMetrics
from .returns import ReturnCalculator
__all__ = [ __all__ = [
"ReturnCalculator", "ReturnCalculator",
"BenchmarkComparison", "BenchmarkComparison",
"PerformanceMetrics", "PerformanceMetrics",
] ]
+2 -5
View File
@@ -3,7 +3,7 @@ Benchmark comparison for calculating abnormal returns (alpha).
""" """
import logging import logging
from datetime import date, timedelta from datetime import date
from decimal import Decimal from decimal import Decimal
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -60,8 +60,7 @@ class BenchmarkComparison:
return None return None
# Calculate return # Calculate return
return_pct = ((end_price - start_price) / start_price) * 100 return ((end_price - start_price) / start_price) * 100
return return_pct
def calculate_abnormal_return( def calculate_abnormal_return(
self, self,
@@ -219,5 +218,3 @@ class BenchmarkComparison:
"benchmark": self.BENCHMARKS.get(benchmark, benchmark), "benchmark": self.BENCHMARKS.get(benchmark, benchmark),
"window_days": window_days, "window_days": window_days,
} }
+16 -35
View File
@@ -4,7 +4,6 @@ Performance metrics and aggregations.
import logging import logging
from collections import defaultdict from collections import defaultdict
from datetime import date
from sqlalchemy import func from sqlalchemy import func
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -52,11 +51,7 @@ class PerformanceMetrics:
if not official: if not official:
return {"error": "Official not found"} return {"error": "Official not found"}
trades = ( trades = self.session.query(Trade).filter(Trade.official_id == official_id).all()
self.session.query(Trade)
.filter(Trade.official_id == official_id)
.all()
)
if not trades: if not trades:
return { return {
@@ -70,9 +65,7 @@ class PerformanceMetrics:
# Calculate returns for all trades # Calculate returns for all trades
returns_data = [] returns_data = []
for trade in trades: for trade in trades:
result = self.benchmark.compare_trade_to_benchmark( result = self.benchmark.compare_trade_to_benchmark(trade, window_days, benchmark)
trade, window_days, benchmark
)
if result: if result:
returns_data.append(result) returns_data.append(result)
@@ -96,9 +89,7 @@ class PerformanceMetrics:
worst_trade = min(returns_data, key=lambda x: x["trade_return"]) worst_trade = min(returns_data, key=lambda x: x["trade_return"])
# Total value traded # Total value traded
total_value = sum( total_value = sum(float(t.value_min or 0) for t in trades if t.value_min)
float(t.value_min or 0) for t in trades if t.value_min
)
return { return {
"name": official.name, "name": official.name,
@@ -154,20 +145,14 @@ class PerformanceMetrics:
List of sector performance dictionaries List of sector performance dictionaries
""" """
# Get all trades with security info # Get all trades with security info
trades = ( trades = self.session.query(Trade).join(Security).all()
self.session.query(Trade)
.join(Security)
.all()
)
# Group by sector # Group by sector
sector_data = defaultdict(list) sector_data = defaultdict(list)
for trade in trades: for trade in trades:
sector = trade.security.sector or "Unknown" sector = trade.security.sector or "Unknown"
result = self.benchmark.compare_trade_to_benchmark( result = self.benchmark.compare_trade_to_benchmark(trade, window_days, benchmark)
trade, window_days, benchmark
)
if result: if result:
sector_data[sector].append(result) sector_data[sector].append(result)
@@ -180,14 +165,16 @@ class PerformanceMetrics:
returns = [d["trade_return"] for d in data] returns = [d["trade_return"] for d in data]
alphas = [d["abnormal_return"] for d in data] alphas = [d["abnormal_return"] for d in data]
results.append({ results.append(
"sector": sector, {
"trade_count": len(data), "sector": sector,
"avg_return": sum(returns) / len(returns), "trade_count": len(data),
"avg_alpha": sum(alphas) / len(alphas), "avg_return": sum(returns) / len(returns),
"win_rate": sum(1 for r in returns if r > 0) / len(returns), "avg_alpha": sum(alphas) / len(alphas),
"beat_market_rate": sum(1 for a in alphas if a > 0) / len(alphas), "win_rate": sum(1 for r in returns if r > 0) / len(returns),
}) "beat_market_rate": sum(1 for a in alphas if a > 0) / len(alphas),
}
)
# Sort by average alpha # Sort by average alpha
results.sort(key=lambda x: x["avg_alpha"], reverse=True) results.sort(key=lambda x: x["avg_alpha"], reverse=True)
@@ -229,11 +216,7 @@ class PerformanceMetrics:
Returns: Returns:
Dictionary with timing statistics Dictionary with timing statistics
""" """
trades = ( trades = self.session.query(Trade).filter(Trade.filing_date.isnot(None)).all()
self.session.query(Trade)
.filter(Trade.filing_date.isnot(None))
.all()
)
if not trades: if not trades:
return {"error": "No trades with disclosure dates"} return {"error": "No trades with disclosure dates"}
@@ -288,5 +271,3 @@ class PerformanceMetrics:
"benchmark": benchmark, "benchmark": benchmark,
**aggregate, **aggregate,
} }
+8 -6
View File
@@ -94,18 +94,20 @@ class ReturnCalculator:
def calculate_multiple_windows( def calculate_multiple_windows(
self, self,
trade: Trade, trade: Trade,
windows: list[int] = [30, 60, 90, 180], windows: list[int] | None = None,
) -> dict[int, dict]: ) -> dict[int, dict]:
""" """
Calculate returns for multiple time windows. Calculate returns for multiple time windows.
Args: Args:
trade: Trade object trade: Trade object
windows: List of window sizes in days windows: List of window sizes in days (defaults to 30/60/90/180)
Returns: Returns:
Dictionary mapping window_days to return metrics Dictionary mapping window_days to return metrics
""" """
if windows is None:
windows = [30, 60, 90, 180]
results = {} results = {}
for window in windows: for window in windows:
result = self.calculate_trade_return(trade, window) result = self.calculate_trade_return(trade, window)
@@ -223,12 +225,13 @@ class ReturnCalculator:
if not prices: if not prices:
return pd.DataFrame() return pd.DataFrame()
# open/high/low are nullable columns; use NaN (pandas-native) when absent.
data = [ data = [
{ {
"date": p.date, "date": p.date,
"open": float(p.open), "open": float(p.open) if p.open is not None else float("nan"),
"high": float(p.high), "high": float(p.high) if p.high is not None else float("nan"),
"low": float(p.low), "low": float(p.low) if p.low is not None else float("nan"),
"close": float(p.close), "close": float(p.close),
"volume": p.volume, "volume": p.volume,
} }
@@ -236,4 +239,3 @@ class ReturnCalculator:
] ]
return pd.DataFrame(data) return pd.DataFrame(data)
+16 -32
View File
@@ -3,17 +3,17 @@ SQLAlchemy ORM models for POTE.
Matches the schema defined in docs/02_data_model.md. Matches the schema defined in docs/02_data_model.md.
""" """
from datetime import date, datetime, timezone from datetime import UTC, date, datetime
from decimal import Decimal from decimal import Decimal
from sqlalchemy import ( from sqlalchemy import (
DECIMAL, DECIMAL,
JSON,
Date, Date,
DateTime, DateTime,
ForeignKey, ForeignKey,
Index, Index,
Integer, Integer,
JSON,
String, String,
Text, Text,
UniqueConstraint, UniqueConstraint,
@@ -35,13 +35,11 @@ class Official(Base):
state: Mapped[str | None] = mapped_column(String(2)) state: Mapped[str | None] = mapped_column(String(2))
bioguide_id: Mapped[str | None] = mapped_column(String(20), unique=True) bioguide_id: Mapped[str | None] = mapped_column(String(20), unique=True)
external_ids: Mapped[str | None] = mapped_column(Text) # JSON blob for other IDs external_ids: Mapped[str | None] = mapped_column(Text) # JSON blob for other IDs
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
DateTime, default=lambda: datetime.now(timezone.utc)
)
updated_at: Mapped[datetime] = mapped_column( updated_at: Mapped[datetime] = mapped_column(
DateTime, DateTime,
default=lambda: datetime.now(timezone.utc), default=lambda: datetime.now(UTC),
onupdate=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(UTC),
) )
# Relationships # Relationships
@@ -63,13 +61,11 @@ class Security(Base):
sector: Mapped[str | None] = mapped_column(String(100)) sector: Mapped[str | None] = mapped_column(String(100))
industry: Mapped[str | None] = mapped_column(String(100)) industry: Mapped[str | None] = mapped_column(String(100))
asset_type: Mapped[str] = mapped_column(String(50), default="stock") # stock, bond, etc. asset_type: Mapped[str] = mapped_column(String(50), default="stock") # stock, bond, etc.
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
DateTime, default=lambda: datetime.now(timezone.utc)
)
updated_at: Mapped[datetime] = mapped_column( updated_at: Mapped[datetime] = mapped_column(
DateTime, DateTime,
default=lambda: datetime.now(timezone.utc), default=lambda: datetime.now(UTC),
onupdate=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(UTC),
) )
# Relationships # Relationships
@@ -107,13 +103,11 @@ class Trade(Base):
# Quality flags (JSON or enum list) # Quality flags (JSON or enum list)
quality_flags: Mapped[str | None] = mapped_column(Text) # e.g., "range_only,delayed_filing" quality_flags: Mapped[str | None] = mapped_column(Text) # e.g., "range_only,delayed_filing"
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
DateTime, default=lambda: datetime.now(timezone.utc)
)
updated_at: Mapped[datetime] = mapped_column( updated_at: Mapped[datetime] = mapped_column(
DateTime, DateTime,
default=lambda: datetime.now(timezone.utc), default=lambda: datetime.now(UTC),
onupdate=lambda: datetime.now(timezone.utc), onupdate=lambda: datetime.now(UTC),
) )
# Relationships # Relationships
@@ -156,9 +150,7 @@ class Price(Base):
adjusted_close: Mapped[Decimal | None] = mapped_column(DECIMAL(15, 4)) adjusted_close: Mapped[Decimal | None] = mapped_column(DECIMAL(15, 4))
source: Mapped[str] = mapped_column(String(50), default="yfinance") source: Mapped[str] = mapped_column(String(50), default="yfinance")
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
DateTime, default=lambda: datetime.now(timezone.utc)
)
# Relationships # Relationships
security: Mapped["Security"] = relationship("Security", back_populates="prices") security: Mapped["Security"] = relationship("Security", back_populates="prices")
@@ -188,9 +180,7 @@ class MetricOfficial(Base):
avg_abnormal_return_1m: Mapped[Decimal | None] = mapped_column(DECIMAL(10, 6)) avg_abnormal_return_1m: Mapped[Decimal | None] = mapped_column(DECIMAL(10, 6))
cluster_label: Mapped[str | None] = mapped_column(String(50)) cluster_label: Mapped[str | None] = mapped_column(String(50))
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
DateTime, default=lambda: datetime.now(timezone.utc)
)
__table_args__ = ( __table_args__ = (
UniqueConstraint("official_id", "calc_date", "calc_version", name="uq_metrics_official"), UniqueConstraint("official_id", "calc_date", "calc_version", name="uq_metrics_official"),
@@ -212,9 +202,7 @@ class MetricTrade(Base):
abnormal_return_1m: Mapped[Decimal | None] = mapped_column(DECIMAL(10, 6)) abnormal_return_1m: Mapped[Decimal | None] = mapped_column(DECIMAL(10, 6))
signal_flags: Mapped[str | None] = mapped_column(Text) # JSON list signal_flags: Mapped[str | None] = mapped_column(Text) # JSON list
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
DateTime, default=lambda: datetime.now(timezone.utc)
)
__table_args__ = ( __table_args__ = (
UniqueConstraint("trade_id", "calc_date", "calc_version", name="uq_metrics_trade"), UniqueConstraint("trade_id", "calc_date", "calc_version", name="uq_metrics_trade"),
@@ -242,18 +230,14 @@ class MarketAlert(Base):
# Metrics at time of alert # Metrics at time of alert
price: Mapped[Decimal | None] = mapped_column(DECIMAL(15, 4)) price: Mapped[Decimal | None] = mapped_column(DECIMAL(15, 4))
volume: Mapped[int | None] = mapped_column(Integer) volume: Mapped[int | None] = mapped_column(Integer)
change_pct: Mapped[Decimal | None] = mapped_column( change_pct: Mapped[Decimal | None] = mapped_column(DECIMAL(10, 4)) # Price change %
DECIMAL(10, 4)
) # Price change %
# Severity scoring # Severity scoring
severity: Mapped[int | None] = mapped_column(Integer) # 1-10 scale severity: Mapped[int | None] = mapped_column(Integer) # 1-10 scale
# Metadata # Metadata
source: Mapped[str] = mapped_column(String(50), default="market_monitor") source: Mapped[str] = mapped_column(String(50), default="market_monitor")
created_at: Mapped[datetime] = mapped_column( created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
DateTime, default=lambda: datetime.now(timezone.utc)
)
# Indexes for efficient queries # Indexes for efficient queries
__table_args__ = ( __table_args__ = (
+5 -5
View File
@@ -14,6 +14,7 @@ import httpx
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
def _default_data_urls() -> tuple[str, ...]: def _default_data_urls() -> tuple[str, ...]:
override = os.environ.get("POTE_HOUSE_DATA_URL", "").strip() override = os.environ.get("POTE_HOUSE_DATA_URL", "").strip()
if override: if override:
@@ -235,10 +236,9 @@ def normalize_transaction_type(txn_type: str) -> str:
if "purchase" in txn_lower or "buy" in txn_lower: if "purchase" in txn_lower or "buy" in txn_lower:
return "buy" return "buy"
elif "sale" in txn_lower or "sell" in txn_lower: if "sale" in txn_lower or "sell" in txn_lower:
return "sell" return "sell"
elif "exchange" in txn_lower: if "exchange" in txn_lower:
return "exchange" return "exchange"
else: # Default to the original, lowercased
# Default to the original, lowercased return txn_lower
return txn_lower
+2 -2
View File
@@ -4,7 +4,7 @@ Fetches daily OHLCV data for securities and stores in the prices table.
""" """
import logging import logging
from datetime import date, datetime, timedelta, timezone from datetime import UTC, date, datetime, timedelta
from decimal import Decimal from decimal import Decimal
import pandas as pd import pandas as pd
@@ -158,7 +158,7 @@ class PriceLoader:
"volume": int(row["volume"]) if pd.notna(row.get("volume")) else None, "volume": int(row["volume"]) if pd.notna(row.get("volume")) else None,
"adjusted_close": None, # We'll compute this later if needed "adjusted_close": None, # We'll compute this later if needed
"source": "yfinance", "source": "yfinance",
"created_at": datetime.now(timezone.utc), "created_at": datetime.now(UTC),
} }
records.append(record) records.append(record)
-1
View File
@@ -9,4 +9,3 @@ from .market_monitor import MarketMonitor
from .pattern_detector import PatternDetector from .pattern_detector import PatternDetector
__all__ = ["MarketMonitor", "AlertManager", "DisclosureCorrelator", "PatternDetector"] __all__ = ["MarketMonitor", "AlertManager", "DisclosureCorrelator", "PatternDetector"]
+19 -24
View File
@@ -4,8 +4,7 @@ Handles alert filtering, formatting, and delivery.
""" """
import logging import logging
from datetime import datetime, timezone from datetime import UTC, datetime
from typing import Any
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -75,21 +74,24 @@ class AlertManager:
Returns: Returns:
HTML formatted alert HTML formatted alert
""" """
severity_class = "high" if (alert.severity or 0) >= 7 else "medium" if (alert.severity or 0) >= 4 else "low" severity_class = (
"high"
if (alert.severity or 0) >= 7
else "medium" if (alert.severity or 0) >= 4 else "low"
)
html = f""" return f"""
<div class="alert {severity_class}"> <div class="alert {severity_class}">
<h3>{alert.ticker} - {alert.alert_type.replace('_', ' ').title()}</h3> <h3>{alert.ticker} - {alert.alert_type.replace('_', ' ').title()}</h3>
<p class="timestamp">{alert.timestamp.strftime('%Y-%m-%d %H:%M:%S')}</p> <p class="timestamp">{alert.timestamp.strftime('%Y-%m-%d %H:%M:%S')}</p>
<p class="severity">Severity: {alert.severity}/10</p> <p class="severity">Severity: {alert.severity}/10</p>
<div class="metrics"> <div class="metrics">
<span>Price: ${float(alert.price):.2f}</span> <span>Price: ${float(alert.price or 0):.2f}</span>
<span>Volume: {alert.volume:,}</span> <span>Volume: {alert.volume:,}</span>
<span>Change: {float(alert.change_pct):+.2f}%</span> <span>Change: {float(alert.change_pct or 0):+.2f}%</span>
</div> </div>
</div> </div>
""" """
return html
def filter_alerts( def filter_alerts(
self, self,
@@ -117,7 +119,7 @@ class AlertManager:
# Filter by ticker # Filter by ticker
if tickers: if tickers:
ticker_set = set(t.upper() for t in tickers) ticker_set = {t.upper() for t in tickers}
filtered = [a for a in filtered if a.ticker.upper() in ticker_set] filtered = [a for a in filtered if a.ticker.upper() in ticker_set]
# Filter by alert type # Filter by alert type
@@ -128,22 +130,21 @@ class AlertManager:
return filtered return filtered
def generate_summary_report( def generate_summary_report(
self, alerts: list[MarketAlert], format: str = "text" self, alerts: list[MarketAlert], output_format: str = "text"
) -> str: ) -> str:
""" """
Generate summary report of alerts. Generate summary report of alerts.
Args: Args:
alerts: List of alerts alerts: List of alerts
format: Output format ('text' or 'html') output_format: Output format ('text' or 'html')
Returns: Returns:
Formatted summary report Formatted summary report
""" """
if format == "html": if output_format == "html":
return self._generate_html_summary(alerts) return self._generate_html_summary(alerts)
else: return self._generate_text_summary(alerts)
return self._generate_text_summary(alerts)
def _generate_text_summary(self, alerts: list[MarketAlert]) -> str: def _generate_text_summary(self, alerts: list[MarketAlert]) -> str:
"""Generate text summary report.""" """Generate text summary report."""
@@ -152,7 +153,7 @@ class AlertManager:
lines = [ lines = [
"=" * 80, "=" * 80,
f" MARKET ACTIVITY ALERTS - {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S')} UTC", f" MARKET ACTIVITY ALERTS - {datetime.now(UTC).strftime('%Y-%m-%d %H:%M:%S')} UTC",
f" {len(alerts)} Alerts", f" {len(alerts)} Alerts",
"=" * 80, "=" * 80,
"", "",
@@ -180,9 +181,7 @@ class AlertManager:
lines.append(f"🎯 {ticker} - {len(ticker_alerts)} alerts (Max Severity: {max_sev}/10)") lines.append(f"🎯 {ticker} - {len(ticker_alerts)} alerts (Max Severity: {max_sev}/10)")
lines.append("" * 80) lines.append("" * 80)
for alert in sorted( for alert in sorted(ticker_alerts, key=lambda a: a.severity or 0, reverse=True):
ticker_alerts, key=lambda a: a.severity or 0, reverse=True
):
lines.append("") lines.append("")
lines.append(self.format_alert_text(alert)) lines.append(self.format_alert_text(alert))
@@ -202,9 +201,7 @@ class AlertManager:
type_counts[alert.alert_type] = type_counts.get(alert.alert_type, 0) + 1 type_counts[alert.alert_type] = type_counts.get(alert.alert_type, 0) + 1
lines.append("\nAlert Types:") lines.append("\nAlert Types:")
for alert_type, count in sorted( for alert_type, count in sorted(type_counts.items(), key=lambda x: x[1], reverse=True):
type_counts.items(), key=lambda x: x[1], reverse=True
):
lines.append(f" {alert_type.replace('_', ' ').title():20s}: {count}") lines.append(f" {alert_type.replace('_', ' ').title():20s}: {count}")
# Top severity alerts # Top severity alerts
@@ -232,8 +229,8 @@ class AlertManager:
".timestamp { color: #666; font-size: 0.9em; }", ".timestamp { color: #666; font-size: 0.9em; }",
".metrics span { margin-right: 20px; }", ".metrics span { margin-right: 20px; }",
"</style></head><body>", "</style></head><body>",
f"<h1>Market Activity Alerts</h1>", "<h1>Market Activity Alerts</h1>",
f"<p><strong>{len(alerts)} Alerts</strong> | {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S')} UTC</p>", f"<p><strong>{len(alerts)} Alerts</strong> | {datetime.now(UTC).strftime('%Y-%m-%d %H:%M:%S')} UTC</p>",
] ]
for alert in sorted(alerts, key=lambda a: a.severity or 0, reverse=True): for alert in sorted(alerts, key=lambda a: a.severity or 0, reverse=True):
@@ -241,5 +238,3 @@ class AlertManager:
html_parts.append("</body></html>") html_parts.append("</body></html>")
return "\n".join(html_parts) return "\n".join(html_parts)
+18 -50
View File
@@ -5,14 +5,13 @@ Calculates timing advantage and suspicious activity scores.
""" """
import logging import logging
from datetime import date, timedelta, timezone from datetime import UTC, date, timedelta
from decimal import Decimal
from typing import Any from typing import Any
from sqlalchemy import and_, func from sqlalchemy import and_
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from pote.db.models import MarketAlert, Official, Security, Trade from pote.db.models import MarketAlert, Security, Trade
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -27,9 +26,7 @@ class DisclosureCorrelator:
"""Initialize disclosure correlator.""" """Initialize disclosure correlator."""
self.session = session self.session = session
def get_alerts_before_trade( def get_alerts_before_trade(self, trade: Trade, lookback_days: int = 30) -> list[MarketAlert]:
self, trade: Trade, lookback_days: int = 30
) -> list[MarketAlert]:
""" """
Get market alerts that occurred BEFORE a trade. Get market alerts that occurred BEFORE a trade.
@@ -50,14 +47,10 @@ class DisclosureCorrelator:
# Convert dates to datetime for comparison # Convert dates to datetime for comparison
from datetime import datetime from datetime import datetime
start_dt = datetime.combine(start_date, datetime.min.time()).replace( start_dt = datetime.combine(start_date, datetime.min.time()).replace(tzinfo=UTC)
tzinfo=timezone.utc end_dt = datetime.combine(end_date, datetime.max.time()).replace(tzinfo=UTC)
)
end_dt = datetime.combine(end_date, datetime.max.time()).replace(
tzinfo=timezone.utc
)
alerts = ( return (
self.session.query(MarketAlert) self.session.query(MarketAlert)
.filter( .filter(
and_( and_(
@@ -70,8 +63,6 @@ class DisclosureCorrelator:
.all() .all()
) )
return alerts
def calculate_timing_score( def calculate_timing_score(
self, trade: Trade, prior_alerts: list[MarketAlert] self, trade: Trade, prior_alerts: list[MarketAlert]
) -> dict[str, Any]: ) -> dict[str, Any]:
@@ -131,8 +122,7 @@ class DisclosureCorrelator:
) )
elif suspicious: elif suspicious:
reason = ( reason = (
f"Trade occurred after {len(prior_alerts)} alerts. " f"Trade occurred after {len(prior_alerts)} alerts. " f"Possible timing advantage."
f"Possible timing advantage."
) )
else: else:
reason = ( reason = (
@@ -170,35 +160,27 @@ class DisclosureCorrelator:
timing_analysis = self.calculate_timing_score(trade, prior_alerts) timing_analysis = self.calculate_timing_score(trade, prior_alerts)
# Build full analysis # Build full analysis
analysis = { return {
"trade_id": trade.id, "trade_id": trade.id,
"official_name": trade.official.name if trade.official else None, "official_name": trade.official.name if trade.official else None,
"ticker": trade.security.ticker if trade.security else None, "ticker": trade.security.ticker if trade.security else None,
"side": trade.side, "side": trade.side,
"transaction_date": str(trade.transaction_date), "transaction_date": str(trade.transaction_date),
"filing_date": str(trade.filing_date) if trade.filing_date else None, "filing_date": str(trade.filing_date) if trade.filing_date else None,
"value_range": f"${float(trade.value_min):,.0f}" "value_range": f"${float(trade.value_min or 0):,.0f}"
+ ( + (f"-${float(trade.value_max):,.0f}" if trade.value_max else "+"),
f"-${float(trade.value_max):,.0f}"
if trade.value_max
else "+"
),
**timing_analysis, **timing_analysis,
"prior_alerts": [ "prior_alerts": [
{ {
"timestamp": str(alert.timestamp), "timestamp": str(alert.timestamp),
"alert_type": alert.alert_type, "alert_type": alert.alert_type,
"severity": alert.severity, "severity": alert.severity,
"days_before_trade": ( "days_before_trade": (trade.transaction_date - alert.timestamp.date()).days,
trade.transaction_date - alert.timestamp.date()
).days,
} }
for alert in prior_alerts for alert in prior_alerts
], ],
} }
return analysis
def analyze_recent_disclosures( def analyze_recent_disclosures(
self, days: int = 7, min_timing_score: float = 50 self, days: int = 7, min_timing_score: float = 50
) -> list[dict[str, Any]]: ) -> list[dict[str, Any]]:
@@ -237,9 +219,7 @@ class DisclosureCorrelator:
f"Found {len(suspicious_trades)} trades with timing score >= {min_timing_score}" f"Found {len(suspicious_trades)} trades with timing score >= {min_timing_score}"
) )
return sorted( return sorted(suspicious_trades, key=lambda x: x["timing_score"], reverse=True)
suspicious_trades, key=lambda x: x["timing_score"], reverse=True
)
def get_official_timing_pattern( def get_official_timing_pattern(
self, official_id: int, lookback_days: int = 365 self, official_id: int, lookback_days: int = 365
@@ -258,9 +238,7 @@ class DisclosureCorrelator:
trades = ( trades = (
self.session.query(Trade) self.session.query(Trade)
.filter( .filter(and_(Trade.official_id == official_id, Trade.transaction_date >= since_date))
and_(Trade.official_id == official_id, Trade.transaction_date >= since_date)
)
.join(Trade.security) .join(Trade.security)
.all() .all()
) )
@@ -285,9 +263,7 @@ class DisclosureCorrelator:
highly_suspicious = sum(1 for a in analyses if a.get("highly_suspicious", False)) highly_suspicious = sum(1 for a in analyses if a.get("highly_suspicious", False))
avg_timing_score = ( avg_timing_score = (
sum(a["timing_score"] for a in analyses) / total_trades sum(a["timing_score"] for a in analyses) / total_trades if total_trades > 0 else 0
if total_trades > 0
else 0
) )
# Determine pattern # Determine pattern
@@ -311,9 +287,7 @@ class DisclosureCorrelator:
"analyses": analyses, "analyses": analyses,
} }
def get_ticker_timing_analysis( def get_ticker_timing_analysis(self, ticker: str, lookback_days: int = 365) -> dict[str, Any]:
self, ticker: str, lookback_days: int = 365
) -> dict[str, Any]:
""" """
Analyze timing patterns for a specific ticker. Analyze timing patterns for a specific ticker.
@@ -329,9 +303,7 @@ class DisclosureCorrelator:
trades = ( trades = (
self.session.query(Trade) self.session.query(Trade)
.join(Trade.security) .join(Trade.security)
.filter( .filter(and_(Security.ticker == ticker, Trade.transaction_date >= since_date))
and_(Security.ticker == ticker, Trade.transaction_date >= since_date)
)
.join(Trade.official) .join(Trade.official)
.all() .all()
) )
@@ -350,10 +322,6 @@ class DisclosureCorrelator:
"trade_count": len(analyses), "trade_count": len(analyses),
"trades_with_alerts": sum(1 for a in analyses if a["alert_count"] > 0), "trades_with_alerts": sum(1 for a in analyses if a["alert_count"] > 0),
"suspicious_count": sum(1 for a in analyses if a["suspicious"]), "suspicious_count": sum(1 for a in analyses if a["suspicious"]),
"avg_timing_score": round( "avg_timing_score": round(sum(a["timing_score"] for a in analyses) / len(analyses), 2),
sum(a["timing_score"] for a in analyses) / len(analyses), 2
),
"analyses": sorted(analyses, key=lambda x: x["timing_score"], reverse=True), "analyses": sorted(analyses, key=lambda x: x["timing_score"], reverse=True),
} }
+9 -15
View File
@@ -4,7 +4,7 @@ Detects unusual activity: volume spikes, price movements, volatility.
""" """
import logging import logging
from datetime import datetime, timedelta, timezone from datetime import UTC, datetime, timedelta
from decimal import Decimal from decimal import Decimal
from typing import Any from typing import Any
@@ -59,7 +59,7 @@ class MarketMonitor:
Returns: Returns:
List of alerts detected List of alerts detected
""" """
alerts = [] alerts: list[dict[str, Any]] = []
try: try:
stock = yf.Ticker(ticker) stock = yf.Ticker(ticker)
@@ -90,7 +90,7 @@ class MarketMonitor:
{ {
"ticker": ticker, "ticker": ticker,
"alert_type": "unusual_volume", "alert_type": "unusual_volume",
"timestamp": datetime.now(timezone.utc), "timestamp": datetime.now(UTC),
"details": { "details": {
"current_volume": int(current_volume), "current_volume": int(current_volume),
"avg_volume": int(avg_volume), "avg_volume": int(avg_volume),
@@ -109,10 +109,8 @@ class MarketMonitor:
alerts.append( alerts.append(
{ {
"ticker": ticker, "ticker": ticker,
"alert_type": "price_spike" "alert_type": "price_spike" if price_change > 0 else "price_drop",
if price_change > 0 "timestamp": datetime.now(UTC),
else "price_drop",
"timestamp": datetime.now(timezone.utc),
"details": { "details": {
"current_price": float(current_price), "current_price": float(current_price),
"prev_price": float(prev["Close"]), "prev_price": float(prev["Close"]),
@@ -129,14 +127,12 @@ class MarketMonitor:
if len(hist) >= 5: if len(hist) >= 5:
recent_volatility = hist["Close"].iloc[-5:].pct_change().abs().mean() recent_volatility = hist["Close"].iloc[-5:].pct_change().abs().mean()
if recent_volatility > avg_price_change * 2 and avg_price_change > 0: if recent_volatility > avg_price_change * 2 and avg_price_change > 0:
severity = min( severity = min(10, int((recent_volatility / avg_price_change) - 1))
10, int((recent_volatility / avg_price_change) - 1)
)
alerts.append( alerts.append(
{ {
"ticker": ticker, "ticker": ticker,
"alert_type": "high_volatility", "alert_type": "high_volatility",
"timestamp": datetime.now(timezone.utc), "timestamp": datetime.now(UTC),
"details": { "details": {
"recent_volatility": round(recent_volatility * 100, 2), "recent_volatility": round(recent_volatility * 100, 2),
"avg_volatility": round(avg_price_change * 100, 2), "avg_volatility": round(avg_price_change * 100, 2),
@@ -227,7 +223,7 @@ class MarketMonitor:
Returns: Returns:
List of MarketAlert objects List of MarketAlert objects
""" """
since = datetime.now(timezone.utc) - timedelta(days=days) since = datetime.now(UTC) - timedelta(days=days)
query = self.session.query(MarketAlert).filter(MarketAlert.timestamp >= since) query = self.session.query(MarketAlert).filter(MarketAlert.timestamp >= since)
@@ -252,7 +248,7 @@ class MarketMonitor:
Returns: Returns:
Dict mapping ticker to alert summary Dict mapping ticker to alert summary
""" """
since = datetime.now(timezone.utc) - timedelta(days=days) since = datetime.now(UTC) - timedelta(days=days)
from sqlalchemy import func from sqlalchemy import func
@@ -278,5 +274,3 @@ class MarketMonitor:
} }
return summary return summary
+13 -30
View File
@@ -5,13 +5,12 @@ Identifies recurring suspicious behavior and trading patterns.
import logging import logging
from datetime import date, timedelta from datetime import date, timedelta
from decimal import Decimal
from typing import Any from typing import Any
from sqlalchemy import and_, func from sqlalchemy import func
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
from pote.db.models import MarketAlert, Official, Security, Trade from pote.db.models import Official, Security, Trade
from pote.monitoring.disclosure_correlator import DisclosureCorrelator from pote.monitoring.disclosure_correlator import DisclosureCorrelator
logger = logging.getLogger(__name__) logger = logging.getLogger(__name__)
@@ -60,9 +59,7 @@ class PatternDetector:
.all() .all()
) )
logger.info( logger.info(f"Analyzing {len(officials_with_trades)} officials with {min_trades}+ trades")
f"Analyzing {len(officials_with_trades)} officials with {min_trades}+ trades"
)
rankings = [] rankings = []
@@ -70,9 +67,7 @@ class PatternDetector:
official_id, name, chamber, party, state, trade_count = official_data official_id, name, chamber, party, state, trade_count = official_data
# Get timing pattern # Get timing pattern
pattern = self.correlator.get_official_timing_pattern( pattern = self.correlator.get_official_timing_pattern(official_id, lookback_days)
official_id, lookback_days
)
if pattern["trade_count"] == 0: if pattern["trade_count"] == 0:
continue continue
@@ -128,9 +123,7 @@ class PatternDetector:
rankings = self.rank_officials_by_timing(lookback_days, min_trades=5) rankings = self.rank_officials_by_timing(lookback_days, min_trades=5)
# Filter for high suspicious rates # Filter for high suspicious rates
offenders = [ offenders = [r for r in rankings if r["suspicious_rate"] >= min_suspicious_rate * 100]
r for r in rankings if r["suspicious_rate"] >= min_suspicious_rate * 100
]
logger.info( logger.info(
f"Found {len(offenders)} officials with {min_suspicious_rate*100}%+ suspicious trades" f"Found {len(offenders)} officials with {min_suspicious_rate*100}%+ suspicious trades"
@@ -155,9 +148,7 @@ class PatternDetector:
# Get tickers with enough trades # Get tickers with enough trades
tickers_with_trades = ( tickers_with_trades = (
self.session.query( self.session.query(Security.ticker, func.count(Trade.id).label("trade_count"))
Security.ticker, func.count(Trade.id).label("trade_count")
)
.join(Trade) .join(Trade)
.filter(Trade.transaction_date >= since_date) .filter(Trade.transaction_date >= since_date)
.group_by(Security.ticker) .group_by(Security.ticker)
@@ -169,10 +160,8 @@ class PatternDetector:
ticker_patterns = [] ticker_patterns = []
for ticker, trade_count in tickers_with_trades: for ticker, _trade_count in tickers_with_trades:
analysis = self.correlator.get_ticker_timing_analysis( analysis = self.correlator.get_ticker_timing_analysis(ticker, lookback_days)
ticker, lookback_days
)
if analysis["trade_count"] == 0: if analysis["trade_count"] == 0:
continue continue
@@ -199,9 +188,7 @@ class PatternDetector:
return ticker_patterns return ticker_patterns
def get_sector_timing_analysis( def get_sector_timing_analysis(self, lookback_days: int = 365) -> dict[str, dict[str, Any]]:
self, lookback_days: int = 365
) -> dict[str, dict[str, Any]]:
""" """
Analyze timing patterns by sector. Analyze timing patterns by sector.
@@ -252,7 +239,7 @@ class PatternDetector:
sector_stats[sector]["suspicious_count"] += 1 sector_stats[sector]["suspicious_count"] += 1
# Calculate averages # Calculate averages
for sector, stats in sector_stats.items(): for stats in sector_stats.values():
if stats["trade_count"] > 0: if stats["trade_count"] > 0:
stats["avg_timing_score"] = round( stats["avg_timing_score"] = round(
stats["total_timing_score"] / stats["trade_count"], 2 stats["total_timing_score"] / stats["trade_count"], 2
@@ -266,9 +253,7 @@ class PatternDetector:
return sector_stats return sector_stats
def get_party_comparison( def get_party_comparison(self, lookback_days: int = 365) -> dict[str, dict[str, Any]]:
self, lookback_days: int = 365
) -> dict[str, dict[str, Any]]:
""" """
Compare timing patterns between political parties. Compare timing patterns between political parties.
@@ -303,7 +288,7 @@ class PatternDetector:
party_stats[party]["officials"].append(ranking) party_stats[party]["officials"].append(ranking)
# Calculate averages # Calculate averages
for party, stats in party_stats.items(): for stats in party_stats.values():
if stats["total_trades"] > 0: if stats["total_trades"] > 0:
stats["avg_timing_score"] = round( stats["avg_timing_score"] = round(
stats["total_timing_score"] / stats["total_trades"], 2 stats["total_timing_score"] / stats["total_trades"], 2
@@ -336,7 +321,7 @@ class PatternDetector:
# Calculate summary statistics # Calculate summary statistics
total_officials = len(official_rankings) total_officials = len(official_rankings)
total_offenders = len(repeat_offenders) total_offenders = len(repeat_offenders)
avg_timing_score = ( avg_timing_score = (
sum(r["avg_timing_score"] for r in official_rankings) / total_officials sum(r["avg_timing_score"] for r in official_rankings) / total_officials
if total_officials > 0 if total_officials > 0
@@ -356,5 +341,3 @@ class PatternDetector:
"sector_analysis": sector_analysis, "sector_analysis": sector_analysis,
"party_comparison": party_comparison, "party_comparison": party_comparison,
} }
-2
View File
@@ -8,5 +8,3 @@ from .email_reporter import EmailReporter
from .report_generator import ReportGenerator from .report_generator import ReportGenerator
__all__ = ["EmailReporter", "ReportGenerator"] __all__ = ["EmailReporter", "ReportGenerator"]
+13 -16
View File
@@ -8,7 +8,6 @@ import logging
import smtplib import smtplib
from email.mime.multipart import MIMEMultipart from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText from email.mime.text import MIMEText
from typing import List, Optional
from pote.config import settings from pote.config import settings
@@ -20,31 +19,30 @@ class EmailReporter:
def __init__( def __init__(
self, self,
smtp_host: Optional[str] = None, smtp_host: str | None = None,
smtp_port: Optional[int] = None, smtp_port: int | None = None,
smtp_user: Optional[str] = None, smtp_user: str | None = None,
smtp_password: Optional[str] = None, smtp_password: str | None = None,
from_email: Optional[str] = None, from_email: str | None = None,
): ):
""" """
Initialize email reporter. Initialize email reporter.
If parameters are not provided, will attempt to use settings from config. If parameters are not provided, will attempt to use settings from config.
""" """
self.smtp_host = smtp_host or getattr(settings, "smtp_host", "localhost") # Settings always defines these fields (with defaults), so direct access is safe.
self.smtp_port = smtp_port or getattr(settings, "smtp_port", 587) self.smtp_host: str = smtp_host or settings.smtp_host
self.smtp_user = smtp_user or getattr(settings, "smtp_user", None) self.smtp_port: int = smtp_port or settings.smtp_port
self.smtp_password = smtp_password or getattr(settings, "smtp_password", None) self.smtp_user: str = smtp_user or settings.smtp_user
self.from_email = from_email or getattr( self.smtp_password: str = smtp_password or settings.smtp_password
settings, "from_email", "pote@localhost" self.from_email: str = from_email or settings.from_email or "pote@localhost"
)
def send_report( def send_report(
self, self,
to_emails: List[str], to_emails: list[str],
subject: str, subject: str,
body_text: str, body_text: str,
body_html: Optional[str] = None, body_html: str | None = None,
) -> bool: ) -> bool:
""" """
Send an email report. Send an email report.
@@ -113,4 +111,3 @@ class EmailReporter:
except Exception as e: except Exception as e:
logger.error(f"SMTP connection test failed: {e}") logger.error(f"SMTP connection test failed: {e}")
return False return False
+54 -39
View File
@@ -6,7 +6,7 @@ Generates formatted reports from database data.
import logging import logging
from datetime import date, datetime, timedelta from datetime import date, datetime, timedelta
from typing import Any, Dict, List, Optional from typing import Any
from sqlalchemy import func from sqlalchemy import func
from sqlalchemy.orm import Session from sqlalchemy.orm import Session
@@ -27,13 +27,15 @@ class ReportGenerator:
self.detector = PatternDetector(session) self.detector = PatternDetector(session)
def generate_daily_summary( def generate_daily_summary(
self, report_date: Optional[date] = None self, report_date: date | None = None, *, lookback_days: int = 1
) -> Dict[str, Any]: ) -> dict[str, Any]:
""" """
Generate a daily summary report. Generate a daily summary report.
Args: Args:
report_date: Date to generate report for (defaults to today) report_date: Date to generate report for (defaults to today)
lookback_days: Include trades filed in the last N days ending on report_date
(defaults to 1, meaning only filings on report_date).
Returns: Returns:
Dictionary containing report data Dictionary containing report data
@@ -41,12 +43,19 @@ class ReportGenerator:
if report_date is None: if report_date is None:
report_date = date.today() report_date = date.today()
if lookback_days < 1:
raise ValueError("lookback_days must be >= 1")
start_of_day = datetime.combine(report_date, datetime.min.time()) start_of_day = datetime.combine(report_date, datetime.min.time())
end_of_day = datetime.combine(report_date, datetime.max.time()) end_of_day = datetime.combine(report_date, datetime.max.time())
# Count new trades filed today filing_start_date = report_date - timedelta(days=lookback_days - 1)
# Trades filed within the lookback window (inclusive)
new_trades = ( new_trades = (
self.session.query(Trade).filter(Trade.filing_date == report_date).all() self.session.query(Trade)
.filter(Trade.filing_date >= filing_start_date, Trade.filing_date <= report_date)
.all()
) )
# Count market alerts today # Count market alerts today
@@ -60,7 +69,7 @@ class ReportGenerator:
) )
# Get high-severity alerts # Get high-severity alerts
critical_alerts = [a for a in new_alerts if a.severity >= 7] critical_alerts = [a for a in new_alerts if (a.severity or 0) >= 7]
# Get suspicious timing matches # Get suspicious timing matches
suspicious_trades = [] suspicious_trades = []
@@ -71,6 +80,8 @@ class ReportGenerator:
return { return {
"date": report_date, "date": report_date,
"filing_start_date": filing_start_date,
"lookback_days": lookback_days,
"new_trades_count": len(new_trades), "new_trades_count": len(new_trades),
"new_trades": [ "new_trades": [
{ {
@@ -99,7 +110,7 @@ class ReportGenerator:
"suspicious_trades": suspicious_trades, "suspicious_trades": suspicious_trades,
} }
def generate_weekly_summary(self) -> Dict[str, Any]: def generate_weekly_summary(self) -> dict[str, Any]:
""" """
Generate a weekly summary report. Generate a weekly summary report.
@@ -110,9 +121,7 @@ class ReportGenerator:
# Most active officials # Most active officials
active_officials = ( active_officials = (
self.session.query( self.session.query(Official.name, func.count(Trade.id).label("trade_count"))
Official.name, func.count(Trade.id).label("trade_count")
)
.join(Trade) .join(Trade)
.filter(Trade.filing_date >= week_ago) .filter(Trade.filing_date >= week_ago)
.group_by(Official.id, Official.name) .group_by(Official.id, Official.name)
@@ -123,9 +132,7 @@ class ReportGenerator:
# Most traded securities # Most traded securities
active_securities = ( active_securities = (
self.session.query( self.session.query(Security.ticker, func.count(Trade.id).label("trade_count"))
Security.ticker, func.count(Trade.id).label("trade_count")
)
.join(Trade) .join(Trade)
.filter(Trade.filing_date >= week_ago) .filter(Trade.filing_date >= week_ago)
.group_by(Security.id, Security.ticker) .group_by(Security.id, Security.ticker)
@@ -135,8 +142,10 @@ class ReportGenerator:
) )
# Get top suspicious patterns # Get top suspicious patterns
# Same fix as branch docs/deploy-email-closed: the old kwarg names never
# existed on identify_repeat_offenders and failed every weekly run.
repeat_offenders = self.detector.identify_repeat_offenders( repeat_offenders = self.detector.identify_repeat_offenders(
days_lookback=7, min_suspicious_trades=2, min_timing_score=40 lookback_days=7, min_suspicious_rate=0.4
) )
return { return {
@@ -146,14 +155,13 @@ class ReportGenerator:
{"name": name, "trade_count": count} for name, count in active_officials {"name": name, "trade_count": count} for name, count in active_officials
], ],
"most_traded_securities": [ "most_traded_securities": [
{"ticker": ticker, "trade_count": count} {"ticker": ticker, "trade_count": count} for ticker, count in active_securities
for ticker, count in active_securities
], ],
"repeat_offenders_count": len(repeat_offenders), "repeat_offenders_count": len(repeat_offenders),
"repeat_offenders": repeat_offenders[:5], # Top 5 "repeat_offenders": repeat_offenders[:5], # Top 5
} }
def format_as_text(self, report_data: Dict[str, Any], report_type: str) -> str: def format_as_text(self, report_data: dict[str, Any], report_type: str) -> str:
""" """
Format report data as plain text. Format report data as plain text.
@@ -166,20 +174,26 @@ class ReportGenerator:
""" """
if report_type == "daily": if report_type == "daily":
return self._format_daily_text(report_data) return self._format_daily_text(report_data)
elif report_type == "weekly": if report_type == "weekly":
return self._format_weekly_text(report_data) return self._format_weekly_text(report_data)
else: return str(report_data)
return str(report_data)
def _format_daily_text(self, data: Dict[str, Any]) -> str: def _format_daily_text(self, data: dict[str, Any]) -> str:
"""Format daily report as plain text.""" """Format daily report as plain text."""
if data.get("lookback_days", 1) > 1:
trades_label = (
f" • Trades Filed (last {data['lookback_days']} days): {data['new_trades_count']}"
)
else:
trades_label = f" • New Trades Filed: {data['new_trades_count']}"
lines = [ lines = [
"=" * 70, "=" * 70,
f"POTE DAILY REPORT - {data['date']}", f"POTE DAILY REPORT - {data['date']}",
"=" * 70, "=" * 70,
"", "",
"📊 SUMMARY", "📊 SUMMARY",
f" • New Trades Filed: {data['new_trades_count']}", trades_label,
f" • Market Alerts: {data['market_alerts_count']}", f" • Market Alerts: {data['market_alerts_count']}",
f" • Critical Alerts (≥7 severity): {data['critical_alerts_count']}", f" • Critical Alerts (≥7 severity): {data['critical_alerts_count']}",
f" • Suspicious Timing Trades: {data['suspicious_trades_count']}", f" • Suspicious Timing Trades: {data['suspicious_trades_count']}",
@@ -227,7 +241,7 @@ class ReportGenerator:
return "\n".join(lines) return "\n".join(lines)
def _format_weekly_text(self, data: Dict[str, Any]) -> str: def _format_weekly_text(self, data: dict[str, Any]) -> str:
"""Format weekly report as plain text.""" """Format weekly report as plain text."""
lines = [ lines = [
"=" * 70, "=" * 70,
@@ -246,9 +260,7 @@ class ReportGenerator:
lines.append(f"{security['ticker']}: {security['trade_count']} trades") lines.append(f"{security['ticker']}: {security['trade_count']} trades")
if data["repeat_offenders"]: if data["repeat_offenders"]:
lines.extend( lines.extend(["", f"⚠️ REPEAT OFFENDERS ({data['repeat_offenders_count']} total)"])
["", f"⚠️ REPEAT OFFENDERS ({data['repeat_offenders_count']} total)"]
)
for offender in data["repeat_offenders"]: for offender in data["repeat_offenders"]:
lines.append( lines.append(
f"{offender['official_name']}: " f"{offender['official_name']}: "
@@ -267,7 +279,7 @@ class ReportGenerator:
return "\n".join(lines) return "\n".join(lines)
def format_as_html(self, report_data: Dict[str, Any], report_type: str) -> str: def format_as_html(self, report_data: dict[str, Any], report_type: str) -> str:
""" """
Format report data as HTML. Format report data as HTML.
@@ -280,13 +292,17 @@ class ReportGenerator:
""" """
if report_type == "daily": if report_type == "daily":
return self._format_daily_html(report_data) return self._format_daily_html(report_data)
elif report_type == "weekly": if report_type == "weekly":
return self._format_weekly_html(report_data) return self._format_weekly_html(report_data)
else: return f"<pre>{report_data}</pre>"
return f"<pre>{report_data}</pre>"
def _format_daily_html(self, data: Dict[str, Any]) -> str: def _format_daily_html(self, data: dict[str, Any]) -> str:
"""Format daily report as HTML.""" """Format daily report as HTML."""
if data.get("lookback_days", 1) > 1:
new_trades_label = f"Trades Filed (last {data['lookback_days']} days):"
else:
new_trades_label = "New Trades:"
html = f""" html = f"""
<html> <html>
<head> <head>
@@ -304,10 +320,10 @@ class ReportGenerator:
</head> </head>
<body> <body>
<h1>POTE Daily Report - {data['date']}</h1> <h1>POTE Daily Report - {data['date']}</h1>
<div class="summary"> <div class="summary">
<h2>📊 Summary</h2> <h2>📊 Summary</h2>
<div class="stat"><strong>New Trades:</strong> {data['new_trades_count']}</div> <div class="stat"><strong>{new_trades_label}</strong> {data['new_trades_count']}</div>
<div class="stat"><strong>Market Alerts:</strong> {data['market_alerts_count']}</div> <div class="stat"><strong>Market Alerts:</strong> {data['market_alerts_count']}</div>
<div class="stat"><strong>Critical Alerts:</strong> {data['critical_alerts_count']}</div> <div class="stat"><strong>Critical Alerts:</strong> {data['critical_alerts_count']}</div>
<div class="stat"><strong>Suspicious Trades:</strong> {data['suspicious_trades_count']}</div> <div class="stat"><strong>Suspicious Trades:</strong> {data['suspicious_trades_count']}</div>
@@ -319,7 +335,7 @@ class ReportGenerator:
for t in data["new_trades"][:10]: for t in data["new_trades"][:10]:
html += f""" html += f"""
<div class="trade"> <div class="trade">
<strong>{t['official']}</strong>: {t['side']} {t['ticker']} <strong>{t['official']}</strong>: {t['side']} {t['ticker']}
(${t['value_min']:,.0f} - ${t['value_max']:,.0f}) on {t['transaction_date']} (${t['value_min']:,.0f} - ${t['value_max']:,.0f}) on {t['transaction_date']}
</div> </div>
""" """
@@ -329,7 +345,7 @@ class ReportGenerator:
for a in data["critical_alerts"][:5]: for a in data["critical_alerts"][:5]:
html += f""" html += f"""
<div class="alert critical"> <div class="alert critical">
<strong>{a['ticker']}</strong>: {a['type']} (severity {a['severity']}) <strong>{a['ticker']}</strong>: {a['type']} (severity {a['severity']})
at {a['timestamp'].strftime('%H:%M:%S')} at {a['timestamp'].strftime('%H:%M:%S')}
</div> </div>
""" """
@@ -354,7 +370,7 @@ class ReportGenerator:
return html return html
def _format_weekly_html(self, data: Dict[str, Any]) -> str: def _format_weekly_html(self, data: dict[str, Any]) -> str:
"""Format weekly report as HTML.""" """Format weekly report as HTML."""
html = f""" html = f"""
<html> <html>
@@ -372,7 +388,7 @@ class ReportGenerator:
<body> <body>
<h1>POTE Weekly Report</h1> <h1>POTE Weekly Report</h1>
<p><strong>Period:</strong> {data['period_start']} to {data['period_end']}</p> <p><strong>Period:</strong> {data['period_start']} to {data['period_end']}</p>
<h2>👥 Most Active Officials</h2> <h2>👥 Most Active Officials</h2>
<table> <table>
<tr><th>Official</th><th>Trade Count</th></tr> <tr><th>Official</th><th>Trade Count</th></tr>
@@ -383,7 +399,7 @@ class ReportGenerator:
html += """ html += """
</table> </table>
<h2>📈 Most Traded Securities</h2> <h2>📈 Most Traded Securities</h2>
<table> <table>
<tr><th>Ticker</th><th>Trade Count</th></tr> <tr><th>Ticker</th><th>Trade Count</th></tr>
@@ -420,4 +436,3 @@ class ReportGenerator:
""" """
return html return html
+2 -2
View File
@@ -22,8 +22,8 @@ def test_db_session() -> Session:
engine = create_engine("sqlite:///:memory:", echo=False) engine = create_engine("sqlite:///:memory:", echo=False)
Base.metadata.create_all(engine) Base.metadata.create_all(engine)
TestSessionLocal = sessionmaker(bind=engine) session_factory = sessionmaker(bind=engine)
session = TestSessionLocal() session = session_factory()
yield session yield session
+21 -17
View File
@@ -1,27 +1,28 @@
"""Tests for analytics module.""" """Tests for analytics module."""
import pytest
from datetime import date, timedelta from datetime import date, timedelta
from decimal import Decimal from decimal import Decimal
from pote.analytics.returns import ReturnCalculator import pytest
from pote.analytics.benchmarks import BenchmarkComparison from pote.analytics.benchmarks import BenchmarkComparison
from pote.analytics.metrics import PerformanceMetrics from pote.analytics.metrics import PerformanceMetrics
from pote.db.models import Official, Security, Trade, Price from pote.analytics.returns import ReturnCalculator
from pote.db.models import Price, Security, Trade
@pytest.fixture @pytest.fixture
def sample_prices(test_db_session, sample_security): def sample_prices(test_db_session, sample_security):
"""Create sample price data for testing.""" """Create sample price data for testing."""
session = test_db_session session = test_db_session
# Add SPY (benchmark) prices # Add SPY (benchmark) prices
spy = Security(ticker="SPY", name="SPDR S&P 500 ETF") spy = Security(ticker="SPY", name="SPDR S&P 500 ETF")
session.add(spy) session.add(spy)
session.flush() session.flush()
base_date = date(2024, 1, 1) base_date = date(2024, 1, 1)
# Create SPY prices # Create SPY prices
for i in range(100): for i in range(100):
price = Price( price = Price(
@@ -34,7 +35,7 @@ def sample_prices(test_db_session, sample_security):
volume=1000000, volume=1000000,
) )
session.add(price) session.add(price)
# Create prices for sample_security (AAPL) # Create prices for sample_security (AAPL)
for i in range(100): for i in range(100):
price = Price( price = Price(
@@ -47,7 +48,7 @@ def sample_prices(test_db_session, sample_security):
volume=50000000, volume=50000000,
) )
session.add(price) session.add(price)
session.commit() session.commit()
return session return session
@@ -80,7 +81,9 @@ def test_return_calculator_basic(test_db_session, sample_official, sample_securi
assert "exit_price" in result assert "exit_price" in result
def test_return_calculator_sell_trade(test_db_session, sample_official, sample_security, sample_prices): def test_return_calculator_sell_trade(
test_db_session, sample_official, sample_security, sample_prices
):
session = test_db_session session = test_db_session
"""Test return calculation for sell trade.""" """Test return calculation for sell trade."""
trade = Trade( trade = Trade(
@@ -130,7 +133,7 @@ def test_benchmark_comparison(test_db_session, sample_official, sample_security,
"""Test benchmark comparison.""" """Test benchmark comparison."""
# Create trade and SPY security # Create trade and SPY security
spy = session.query(Security).filter_by(ticker="SPY").first() spy = session.query(Security).filter_by(ticker="SPY").first()
trade = Trade( trade = Trade(
official_id=sample_official.id, official_id=sample_official.id,
security_id=spy.id, security_id=spy.id,
@@ -154,12 +157,14 @@ def test_benchmark_comparison(test_db_session, sample_official, sample_security,
assert "beat_market" in result assert "beat_market" in result
def test_performance_metrics_official(test_db_session, sample_official, sample_security, sample_prices): def test_performance_metrics_official(
test_db_session, sample_official, sample_security, sample_prices
):
session = test_db_session session = test_db_session
"""Test official performance metrics.""" """Test official performance metrics."""
# Create multiple trades # Create multiple trades
spy = session.query(Security).filter_by(ticker="SPY").first() spy = session.query(Security).filter_by(ticker="SPY").first()
for i in range(3): for i in range(3):
trade = Trade( trade = Trade(
official_id=sample_official.id, official_id=sample_official.id,
@@ -171,7 +176,7 @@ def test_performance_metrics_official(test_db_session, sample_official, sample_s
value_max=Decimal("50000"), value_max=Decimal("50000"),
) )
session.add(trade) session.add(trade)
session.commit() session.commit()
# Get performance metrics # Get performance metrics
@@ -187,7 +192,7 @@ def test_multiple_windows(test_db_session, sample_official, sample_security, sam
session = test_db_session session = test_db_session
"""Test calculating returns for multiple windows.""" """Test calculating returns for multiple windows."""
spy = session.query(Security).filter_by(ticker="SPY").first() spy = session.query(Security).filter_by(ticker="SPY").first()
trade = Trade( trade = Trade(
official_id=sample_official.id, official_id=sample_official.id,
security_id=spy.id, security_id=spy.id,
@@ -231,7 +236,7 @@ def test_sector_analysis(test_db_session, sample_official, sample_prices):
value_max=Decimal("50000"), value_max=Decimal("50000"),
) )
session.add(trade) session.add(trade)
session.commit() session.commit()
metrics = PerformanceMetrics(session) metrics = PerformanceMetrics(session)
@@ -257,7 +262,7 @@ def test_timing_analysis(test_db_session, sample_official, sample_security):
value_max=Decimal("50000"), value_max=Decimal("50000"),
) )
session.add(trade) session.add(trade)
session.commit() session.commit()
metrics = PerformanceMetrics(session) metrics = PerformanceMetrics(session)
@@ -265,4 +270,3 @@ def test_timing_analysis(test_db_session, sample_official, sample_security):
assert "avg_disclosure_lag_days" in timing assert "avg_disclosure_lag_days" in timing
assert timing["avg_disclosure_lag_days"] > 0 assert timing["avg_disclosure_lag_days"] > 0
+61 -57
View File
@@ -1,13 +1,14 @@
"""Integration tests for analytics with real-ish data.""" """Integration tests for analytics with real-ish data."""
import pytest
from datetime import date, timedelta from datetime import date, timedelta
from decimal import Decimal from decimal import Decimal
from pote.analytics.returns import ReturnCalculator import pytest
from pote.analytics.benchmarks import BenchmarkComparison from pote.analytics.benchmarks import BenchmarkComparison
from pote.analytics.metrics import PerformanceMetrics from pote.analytics.metrics import PerformanceMetrics
from pote.db.models import Official, Security, Trade, Price from pote.analytics.returns import ReturnCalculator
from pote.db.models import Official, Price, Security, Trade
@pytest.fixture @pytest.fixture
@@ -39,13 +40,13 @@ def full_test_data(test_db_session):
# Create price data for NVDA (upward trend) # Create price data for NVDA (upward trend)
base_date = date(2024, 1, 1) base_date = date(2024, 1, 1)
nvda_base_price = Decimal("495.00") nvda_base_price = Decimal("495.00")
for i in range(120): for i in range(120):
current_date = base_date + timedelta(days=i) current_date = base_date + timedelta(days=i)
# Simulate upward trend: +0.5% per day on average # Simulate upward trend: +0.5% per day on average
price_change = Decimal(i) * Decimal("2.50") # ~50% gain over 120 days price_change = Decimal(i) * Decimal("2.50") # ~50% gain over 120 days
current_price = nvda_base_price + price_change current_price = nvda_base_price + price_change
price = Price( price = Price(
security_id=nvda.id, security_id=nvda.id,
date=current_date, date=current_date,
@@ -59,12 +60,12 @@ def full_test_data(test_db_session):
# Create price data for SPY (slower upward trend - ~10% over 120 days) # Create price data for SPY (slower upward trend - ~10% over 120 days)
spy_base_price = Decimal("450.00") spy_base_price = Decimal("450.00")
for i in range(120): for i in range(120):
current_date = base_date + timedelta(days=i) current_date = base_date + timedelta(days=i)
price_change = Decimal(i) * Decimal("0.35") price_change = Decimal(i) * Decimal("0.35")
current_price = spy_base_price + price_change current_price = spy_base_price + price_change
price = Price( price = Price(
security_id=spy.id, security_id=spy.id,
date=current_date, date=current_date,
@@ -88,7 +89,7 @@ def full_test_data(test_db_session):
value_min=Decimal("15001"), value_min=Decimal("15001"),
value_max=Decimal("50000"), value_max=Decimal("50000"),
) )
# Tuberville buys NVDA later (still good but less alpha) # Tuberville buys NVDA later (still good but less alpha)
trade2 = Trade( trade2 = Trade(
official_id=tuberville.id, official_id=tuberville.id,
@@ -115,22 +116,24 @@ def test_return_calculation_with_real_data(test_db_session, full_test_data):
session = test_db_session session = test_db_session
"""Test return calculation with realistic price data.""" """Test return calculation with realistic price data."""
calculator = ReturnCalculator(session) calculator = ReturnCalculator(session)
# Get Pelosi's NVDA trade # Get Pelosi's NVDA trade
trade = full_test_data["trades"][0] trade = full_test_data["trades"][0]
# Calculate 90-day return # Calculate 90-day return
result = calculator.calculate_trade_return(trade, window_days=90) result = calculator.calculate_trade_return(trade, window_days=90)
assert result is not None, "Should calculate return with available data" assert result is not None, "Should calculate return with available data"
assert result["ticker"] == "NVDA" assert result["ticker"] == "NVDA"
assert result["window_days"] == 90 assert result["window_days"] == 90
assert result["return_pct"] > 0, "NVDA should have positive return" assert result["return_pct"] > 0, "NVDA should have positive return"
# Entry around day 15, exit around day 105 # Entry around day 15, exit around day 105
# Expected return: (720 - 532.5) / 532.5 = ~35% # Expected return: (720 - 532.5) / 532.5 = ~35%
assert 30 < float(result["return_pct"]) < 50, f"Expected ~35% return, got {result['return_pct']}" assert (
30 < float(result["return_pct"]) < 50
), f"Expected ~35% return, got {result['return_pct']}"
print(f"\n✅ NVDA 90-day return: {result['return_pct']:.2f}%") print(f"\n✅ NVDA 90-day return: {result['return_pct']:.2f}%")
print(f" Entry: ${result['entry_price']} on {result['transaction_date']}") print(f" Entry: ${result['entry_price']} on {result['transaction_date']}")
print(f" Exit: ${result['exit_price']} on {result['exit_date']}") print(f" Exit: ${result['exit_price']} on {result['exit_date']}")
@@ -140,22 +143,22 @@ def test_benchmark_comparison_with_real_data(test_db_session, full_test_data):
session = test_db_session session = test_db_session
"""Test benchmark comparison with SPY.""" """Test benchmark comparison with SPY."""
benchmark = BenchmarkComparison(session) benchmark = BenchmarkComparison(session)
# Get Pelosi's trade # Get Pelosi's trade
trade = full_test_data["trades"][0] trade = full_test_data["trades"][0]
# Compare to SPY # Compare to SPY
result = benchmark.compare_trade_to_benchmark(trade, window_days=90, benchmark="SPY") result = benchmark.compare_trade_to_benchmark(trade, window_days=90, benchmark="SPY")
assert result is not None assert result is not None
assert result["ticker"] == "NVDA" assert result["ticker"] == "NVDA"
assert result["benchmark"] == "SPY" assert result["benchmark"] == "SPY"
# NVDA should beat SPY significantly # NVDA should beat SPY significantly
assert result["beat_market"] is True assert result["beat_market"] is True
assert float(result["abnormal_return"]) > 10, "NVDA should have strong alpha vs SPY" assert float(result["abnormal_return"]) > 10, "NVDA should have strong alpha vs SPY"
print(f"\n✅ Benchmark Comparison:") print("\n✅ Benchmark Comparison:")
print(f" NVDA Return: {result['trade_return']:.2f}%") print(f" NVDA Return: {result['trade_return']:.2f}%")
print(f" SPY Return: {result['benchmark_return']:.2f}%") print(f" SPY Return: {result['benchmark_return']:.2f}%")
print(f" Alpha: {result['abnormal_return']:+.2f}%") print(f" Alpha: {result['abnormal_return']:+.2f}%")
@@ -165,21 +168,21 @@ def test_official_performance_summary(test_db_session, full_test_data):
session = test_db_session session = test_db_session
"""Test official performance aggregation.""" """Test official performance aggregation."""
metrics = PerformanceMetrics(session) metrics = PerformanceMetrics(session)
pelosi = full_test_data["officials"][0] pelosi = full_test_data["officials"][0]
# Get performance summary # Get performance summary
perf = metrics.official_performance(pelosi.id, window_days=90) perf = metrics.official_performance(pelosi.id, window_days=90)
assert perf["name"] == "Nancy Pelosi" assert perf["name"] == "Nancy Pelosi"
assert perf["total_trades"] >= 1 assert perf["total_trades"] >= 1
if perf.get("trades_analyzed", 0) > 0: if perf.get("trades_analyzed", 0) > 0:
assert "avg_return" in perf assert "avg_return" in perf
assert "avg_alpha" in perf assert "avg_alpha" in perf
assert "win_rate" in perf assert "win_rate" in perf
assert perf["win_rate"] >= 0 and perf["win_rate"] <= 1 assert perf["win_rate"] >= 0 and perf["win_rate"] <= 1
print(f"\n{perf['name']} Performance:") print(f"\n{perf['name']} Performance:")
print(f" Total Trades: {perf['total_trades']}") print(f" Total Trades: {perf['total_trades']}")
print(f" Average Return: {perf['avg_return']:.2f}%") print(f" Average Return: {perf['avg_return']:.2f}%")
@@ -191,17 +194,17 @@ def test_multiple_windows(test_db_session, full_test_data):
session = test_db_session session = test_db_session
"""Test calculating multiple time windows.""" """Test calculating multiple time windows."""
calculator = ReturnCalculator(session) calculator = ReturnCalculator(session)
trade = full_test_data["trades"][0] trade = full_test_data["trades"][0]
# Calculate for 30, 60, 90 days # Calculate for 30, 60, 90 days
results = calculator.calculate_multiple_windows(trade, windows=[30, 60, 90]) results = calculator.calculate_multiple_windows(trade, windows=[30, 60, 90])
assert len(results) == 3, "Should calculate all three windows" assert len(results) == 3, "Should calculate all three windows"
# Returns should generally increase with longer windows (given upward trend) # Returns should generally increase with longer windows (given upward trend)
if 30 in results and 90 in results: if 30 in results and 90 in results:
print(f"\n✅ Multiple Windows:") print("\n✅ Multiple Windows:")
for window in [30, 60, 90]: for window in [30, 60, 90]:
if window in results: if window in results:
print(f" {window:3d} days: {results[window]['return_pct']:+7.2f}%") print(f" {window:3d} days: {results[window]['return_pct']:+7.2f}%")
@@ -211,13 +214,13 @@ def test_top_performers(test_db_session, full_test_data):
session = test_db_session session = test_db_session
"""Test top performer ranking.""" """Test top performer ranking."""
metrics = PerformanceMetrics(session) metrics = PerformanceMetrics(session)
top = metrics.top_performers(window_days=90, limit=5) top = metrics.top_performers(window_days=90, limit=5)
assert isinstance(top, list) assert isinstance(top, list)
assert len(top) > 0 assert len(top) > 0
print(f"\n✅ Top Performers:") print("\n✅ Top Performers:")
for i, perf in enumerate(top, 1): for i, perf in enumerate(top, 1):
if perf.get("trades_analyzed", 0) > 0: if perf.get("trades_analyzed", 0) > 0:
print(f" {i}. {perf['name']:20s} | Alpha: {perf['avg_alpha']:+6.2f}%") print(f" {i}. {perf['name']:20s} | Alpha: {perf['avg_alpha']:+6.2f}%")
@@ -227,18 +230,18 @@ def test_system_statistics(test_db_session, full_test_data):
session = test_db_session session = test_db_session
"""Test system-wide statistics.""" """Test system-wide statistics."""
metrics = PerformanceMetrics(session) metrics = PerformanceMetrics(session)
stats = metrics.summary_statistics(window_days=90) stats = metrics.summary_statistics(window_days=90)
assert stats["total_officials"] >= 2 assert stats["total_officials"] >= 2
assert stats["total_trades"] >= 2 assert stats["total_trades"] >= 2
assert stats["total_securities"] >= 2 assert stats["total_securities"] >= 2
print(f"\n✅ System Statistics:") print("\n✅ System Statistics:")
print(f" Officials: {stats['total_officials']}") print(f" Officials: {stats['total_officials']}")
print(f" Trades: {stats['total_trades']}") print(f" Trades: {stats['total_trades']}")
print(f" Securities: {stats['total_securities']}") print(f" Securities: {stats['total_securities']}")
if stats.get("avg_alpha") is not None: if stats.get("avg_alpha") is not None:
print(f" Avg Alpha: {stats['avg_alpha']:+.2f}%") print(f" Avg Alpha: {stats['avg_alpha']:+.2f}%")
print(f" Beat Market: {stats['beat_market_rate']:.1%}") print(f" Beat Market: {stats['beat_market_rate']:.1%}")
@@ -248,13 +251,13 @@ def test_disclosure_timing(test_db_session, full_test_data):
session = test_db_session session = test_db_session
"""Test disclosure lag analysis.""" """Test disclosure lag analysis."""
metrics = PerformanceMetrics(session) metrics = PerformanceMetrics(session)
timing = metrics.timing_analysis() timing = metrics.timing_analysis()
assert "avg_disclosure_lag_days" in timing assert "avg_disclosure_lag_days" in timing
assert timing["avg_disclosure_lag_days"] > 0 assert timing["avg_disclosure_lag_days"] > 0
print(f"\n✅ Disclosure Timing:") print("\n✅ Disclosure Timing:")
print(f" Average Lag: {timing['avg_disclosure_lag_days']:.1f} days") print(f" Average Lag: {timing['avg_disclosure_lag_days']:.1f} days")
print(f" Median Lag: {timing['median_disclosure_lag_days']} days") print(f" Median Lag: {timing['median_disclosure_lag_days']} days")
@@ -263,25 +266,27 @@ def test_sector_analysis(test_db_session, full_test_data):
session = test_db_session session = test_db_session
"""Test sector-level analysis.""" """Test sector-level analysis."""
metrics = PerformanceMetrics(session) metrics = PerformanceMetrics(session)
sectors = metrics.sector_analysis(window_days=90) sectors = metrics.sector_analysis(window_days=90)
assert isinstance(sectors, list) assert isinstance(sectors, list)
if sectors: if sectors:
print(f"\n✅ Sector Analysis:") print("\n✅ Sector Analysis:")
for s in sectors: for s in sectors:
print(f" {s['sector']:15s} | {s['trade_count']} trades | Alpha: {s['avg_alpha']:+6.2f}%") print(
f" {s['sector']:15s} | {s['trade_count']} trades | Alpha: {s['avg_alpha']:+6.2f}%"
)
def test_edge_case_missing_exit_price(test_db_session, full_test_data): def test_edge_case_missing_exit_price(test_db_session, full_test_data):
session = test_db_session session = test_db_session
"""Test handling of trade with no exit price available.""" """Test handling of trade with no exit price available."""
calculator = ReturnCalculator(session) calculator = ReturnCalculator(session)
nvda = session.query(Security).filter_by(ticker="NVDA").first() nvda = session.query(Security).filter_by(ticker="NVDA").first()
pelosi = session.query(Official).filter_by(name="Nancy Pelosi").first() pelosi = session.query(Official).filter_by(name="Nancy Pelosi").first()
# Create trade with transaction date far in future (no exit price) # Create trade with transaction date far in future (no exit price)
future_trade = Trade( future_trade = Trade(
official_id=pelosi.id, official_id=pelosi.id,
@@ -294,9 +299,9 @@ def test_edge_case_missing_exit_price(test_db_session, full_test_data):
) )
session.add(future_trade) session.add(future_trade)
session.commit() session.commit()
result = calculator.calculate_trade_return(future_trade, window_days=90) result = calculator.calculate_trade_return(future_trade, window_days=90)
assert result is None, "Should return None when price data unavailable" assert result is None, "Should return None when price data unavailable"
print("\n✅ Correctly handles missing price data") print("\n✅ Correctly handles missing price data")
@@ -305,10 +310,10 @@ def test_sell_trade_logic(test_db_session, full_test_data):
session = test_db_session session = test_db_session
"""Test that sell trades have inverted return logic.""" """Test that sell trades have inverted return logic."""
calculator = ReturnCalculator(session) calculator = ReturnCalculator(session)
nvda = session.query(Security).filter_by(ticker="NVDA").first() nvda = session.query(Security).filter_by(ticker="NVDA").first()
pelosi = session.query(Official).filter_by(name="Nancy Pelosi").first() pelosi = session.query(Official).filter_by(name="Nancy Pelosi").first()
# Create sell trade during uptrend (should show negative return) # Create sell trade during uptrend (should show negative return)
sell_trade = Trade( sell_trade = Trade(
official_id=pelosi.id, official_id=pelosi.id,
@@ -321,11 +326,10 @@ def test_sell_trade_logic(test_db_session, full_test_data):
) )
session.add(sell_trade) session.add(sell_trade)
session.commit() session.commit()
result = calculator.calculate_trade_return(sell_trade, window_days=90) result = calculator.calculate_trade_return(sell_trade, window_days=90)
if result: if result:
# Selling during uptrend = negative return # Selling during uptrend = negative return
assert result["return_pct"] < 0, "Sell during uptrend should show negative return" assert result["return_pct"] < 0, "Sell during uptrend should show negative return"
print(f"\n✅ Sell trade return correctly inverted: {result['return_pct']:.2f}%") print(f"\n✅ Sell trade return correctly inverted: {result['return_pct']:.2f}%")
+74 -75
View File
@@ -1,24 +1,25 @@
"""Tests for disclosure correlation module.""" """Tests for disclosure correlation module."""
import pytest from datetime import UTC, date, datetime
from datetime import date, datetime, timedelta, timezone
from decimal import Decimal from decimal import Decimal
import pytest
from pote.db.models import MarketAlert, Official, Security, Trade
from pote.monitoring.disclosure_correlator import DisclosureCorrelator from pote.monitoring.disclosure_correlator import DisclosureCorrelator
from pote.db.models import Official, Security, Trade, MarketAlert
@pytest.fixture @pytest.fixture
def trade_with_alerts(test_db_session): def trade_with_alerts(test_db_session):
"""Create a trade with prior market alerts.""" """Create a trade with prior market alerts."""
session = test_db_session session = test_db_session
# Create official and security # Create official and security
pelosi = Official(name="Nancy Pelosi", chamber="House", party="Democrat", state="CA") pelosi = Official(name="Nancy Pelosi", chamber="House", party="Democrat", state="CA")
nvda = Security(ticker="NVDA", name="NVIDIA Corporation", sector="Technology") nvda = Security(ticker="NVDA", name="NVIDIA Corporation", sector="Technology")
session.add_all([pelosi, nvda]) session.add_all([pelosi, nvda])
session.flush() session.flush()
# Create trade on Jan 15 # Create trade on Jan 15
trade = Trade( trade = Trade(
official_id=pelosi.id, official_id=pelosi.id,
@@ -32,13 +33,13 @@ def trade_with_alerts(test_db_session):
) )
session.add(trade) session.add(trade)
session.flush() session.flush()
# Create alerts BEFORE trade (suspicious) # Create alerts BEFORE trade (suspicious)
alerts = [ alerts = [
MarketAlert( MarketAlert(
ticker="NVDA", ticker="NVDA",
alert_type="unusual_volume", alert_type="unusual_volume",
timestamp=datetime(2024, 1, 10, 10, 30, tzinfo=timezone.utc), # 5 days before timestamp=datetime(2024, 1, 10, 10, 30, tzinfo=UTC), # 5 days before
details={"multiplier": 3.5}, details={"multiplier": 3.5},
price=Decimal("490.00"), price=Decimal("490.00"),
volume=100000000, volume=100000000,
@@ -48,7 +49,7 @@ def trade_with_alerts(test_db_session):
MarketAlert( MarketAlert(
ticker="NVDA", ticker="NVDA",
alert_type="price_spike", alert_type="price_spike",
timestamp=datetime(2024, 1, 12, 14, 15, tzinfo=timezone.utc), # 3 days before timestamp=datetime(2024, 1, 12, 14, 15, tzinfo=UTC), # 3 days before
details={"change_pct": 5.5}, details={"change_pct": 5.5},
price=Decimal("505.00"), price=Decimal("505.00"),
volume=85000000, volume=85000000,
@@ -58,7 +59,7 @@ def trade_with_alerts(test_db_session):
MarketAlert( MarketAlert(
ticker="NVDA", ticker="NVDA",
alert_type="high_volatility", alert_type="high_volatility",
timestamp=datetime(2024, 1, 14, 16, 20, tzinfo=timezone.utc), # 1 day before timestamp=datetime(2024, 1, 14, 16, 20, tzinfo=UTC), # 1 day before
details={"multiplier": 2.5}, details={"multiplier": 2.5},
price=Decimal("510.00"), price=Decimal("510.00"),
volume=90000000, volume=90000000,
@@ -68,7 +69,7 @@ def trade_with_alerts(test_db_session):
] ]
session.add_all(alerts) session.add_all(alerts)
session.commit() session.commit()
return { return {
"trade": trade, "trade": trade,
"official": pelosi, "official": pelosi,
@@ -81,12 +82,12 @@ def trade_with_alerts(test_db_session):
def trade_without_alerts(test_db_session): def trade_without_alerts(test_db_session):
"""Create a trade without prior alerts (clean).""" """Create a trade without prior alerts (clean)."""
session = test_db_session session = test_db_session
official = Official(name="John Smith", chamber="House", party="Republican", state="TX") official = Official(name="John Smith", chamber="House", party="Republican", state="TX")
security = Security(ticker="MSFT", name="Microsoft Corporation", sector="Technology") security = Security(ticker="MSFT", name="Microsoft Corporation", sector="Technology")
session.add_all([official, security]) session.add_all([official, security])
session.flush() session.flush()
trade = Trade( trade = Trade(
official_id=official.id, official_id=official.id,
security_id=security.id, security_id=security.id,
@@ -98,7 +99,7 @@ def trade_without_alerts(test_db_session):
) )
session.add(trade) session.add(trade)
session.commit() session.commit()
return { return {
"trade": trade, "trade": trade,
"official": official, "official": official,
@@ -110,12 +111,12 @@ def test_get_alerts_before_trade(test_db_session, trade_with_alerts):
"""Test retrieving alerts before a trade.""" """Test retrieving alerts before a trade."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
trade = trade_with_alerts["trade"] trade = trade_with_alerts["trade"]
# Get alerts before trade # Get alerts before trade
prior_alerts = correlator.get_alerts_before_trade(trade, lookback_days=30) prior_alerts = correlator.get_alerts_before_trade(trade, lookback_days=30)
assert len(prior_alerts) == 3 assert len(prior_alerts) == 3
assert all(alert.ticker == "NVDA" for alert in prior_alerts) assert all(alert.ticker == "NVDA" for alert in prior_alerts)
assert all(alert.timestamp.date() < trade.transaction_date for alert in prior_alerts) assert all(alert.timestamp.date() < trade.transaction_date for alert in prior_alerts)
@@ -125,11 +126,11 @@ def test_get_alerts_before_trade_no_alerts(test_db_session, trade_without_alerts
"""Test retrieving alerts when none exist.""" """Test retrieving alerts when none exist."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
trade = trade_without_alerts["trade"] trade = trade_without_alerts["trade"]
prior_alerts = correlator.get_alerts_before_trade(trade, lookback_days=30) prior_alerts = correlator.get_alerts_before_trade(trade, lookback_days=30)
assert len(prior_alerts) == 0 assert len(prior_alerts) == 0
@@ -137,12 +138,12 @@ def test_calculate_timing_score_high_suspicion(test_db_session, trade_with_alert
"""Test timing score calculation for suspicious trade.""" """Test timing score calculation for suspicious trade."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
trade = trade_with_alerts["trade"] trade = trade_with_alerts["trade"]
alerts = trade_with_alerts["alerts"] alerts = trade_with_alerts["alerts"]
timing_analysis = correlator.calculate_timing_score(trade, alerts) timing_analysis = correlator.calculate_timing_score(trade, alerts)
assert timing_analysis["timing_score"] > 60, "Should be suspicious with 3 alerts" assert timing_analysis["timing_score"] > 60, "Should be suspicious with 3 alerts"
assert timing_analysis["suspicious"] is True assert timing_analysis["suspicious"] is True
assert timing_analysis["alert_count"] == 3 assert timing_analysis["alert_count"] == 3
@@ -155,13 +156,13 @@ def test_calculate_timing_score_no_alerts(test_db_session):
"""Test timing score with no prior alerts.""" """Test timing score with no prior alerts."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
# Create minimal trade # Create minimal trade
official = Official(name="Test", chamber="House", party="Democrat", state="CA") official = Official(name="Test", chamber="House", party="Democrat", state="CA")
security = Security(ticker="TEST", name="Test Corp") security = Security(ticker="TEST", name="Test Corp")
session.add_all([official, security]) session.add_all([official, security])
session.flush() session.flush()
trade = Trade( trade = Trade(
official_id=official.id, official_id=official.id,
security_id=security.id, security_id=security.id,
@@ -172,9 +173,9 @@ def test_calculate_timing_score_no_alerts(test_db_session):
) )
session.add(trade) session.add(trade)
session.commit() session.commit()
timing_analysis = correlator.calculate_timing_score(trade, []) timing_analysis = correlator.calculate_timing_score(trade, [])
assert timing_analysis["timing_score"] == 0 assert timing_analysis["timing_score"] == 0
assert timing_analysis["suspicious"] is False assert timing_analysis["suspicious"] is False
assert timing_analysis["alert_count"] == 0 assert timing_analysis["alert_count"] == 0
@@ -184,13 +185,13 @@ def test_calculate_timing_score_factors(test_db_session):
"""Test that timing score considers all factors correctly.""" """Test that timing score considers all factors correctly."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
# Create trade # Create trade
official = Official(name="Test", chamber="House", party="Democrat", state="CA") official = Official(name="Test", chamber="House", party="Democrat", state="CA")
security = Security(ticker="TEST", name="Test Corp") security = Security(ticker="TEST", name="Test Corp")
session.add_all([official, security]) session.add_all([official, security])
session.flush() session.flush()
trade_date = date(2024, 1, 15) trade_date = date(2024, 1, 15)
trade = Trade( trade = Trade(
official_id=official.id, official_id=official.id,
@@ -202,47 +203,47 @@ def test_calculate_timing_score_factors(test_db_session):
) )
session.add(trade) session.add(trade)
session.flush() session.flush()
# Test with low severity alerts (should have lower score) # Test with low severity alerts (should have lower score)
low_sev_alerts = [ low_sev_alerts = [
MarketAlert( MarketAlert(
ticker="TEST", ticker="TEST",
alert_type="test", alert_type="test",
timestamp=datetime(2024, 1, 10, 12, 0, tzinfo=timezone.utc), timestamp=datetime(2024, 1, 10, 12, 0, tzinfo=UTC),
severity=3, severity=3,
), ),
MarketAlert( MarketAlert(
ticker="TEST", ticker="TEST",
alert_type="test", alert_type="test",
timestamp=datetime(2024, 1, 11, 12, 0, tzinfo=timezone.utc), timestamp=datetime(2024, 1, 11, 12, 0, tzinfo=UTC),
severity=4, severity=4,
), ),
] ]
session.add_all(low_sev_alerts) session.add_all(low_sev_alerts)
session.commit() session.commit()
low_score = correlator.calculate_timing_score(trade, low_sev_alerts) low_score = correlator.calculate_timing_score(trade, low_sev_alerts)
# Test with high severity alerts (should have higher score) # Test with high severity alerts (should have higher score)
high_sev_alerts = [ high_sev_alerts = [
MarketAlert( MarketAlert(
ticker="TEST", ticker="TEST",
alert_type="test", alert_type="test",
timestamp=datetime(2024, 1, 13, 12, 0, tzinfo=timezone.utc), # Recent timestamp=datetime(2024, 1, 13, 12, 0, tzinfo=UTC), # Recent
severity=9, severity=9,
), ),
MarketAlert( MarketAlert(
ticker="TEST", ticker="TEST",
alert_type="test", alert_type="test",
timestamp=datetime(2024, 1, 14, 12, 0, tzinfo=timezone.utc), # Very recent timestamp=datetime(2024, 1, 14, 12, 0, tzinfo=UTC), # Very recent
severity=8, severity=8,
), ),
] ]
session.add_all(high_sev_alerts) session.add_all(high_sev_alerts)
session.commit() session.commit()
high_score = correlator.calculate_timing_score(trade, high_sev_alerts) high_score = correlator.calculate_timing_score(trade, high_sev_alerts)
# High severity + recent should score higher # High severity + recent should score higher
assert high_score["timing_score"] > low_score["timing_score"] assert high_score["timing_score"] > low_score["timing_score"]
assert high_score["recent_alert_count"] > 0 assert high_score["recent_alert_count"] > 0
@@ -253,11 +254,11 @@ def test_analyze_trade_full(test_db_session, trade_with_alerts):
"""Test complete trade analysis.""" """Test complete trade analysis."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
trade = trade_with_alerts["trade"] trade = trade_with_alerts["trade"]
analysis = correlator.analyze_trade(trade) analysis = correlator.analyze_trade(trade)
# Check all required fields # Check all required fields
assert analysis["trade_id"] == trade.id assert analysis["trade_id"] == trade.id
assert analysis["official_name"] == "Nancy Pelosi" assert analysis["official_name"] == "Nancy Pelosi"
@@ -267,7 +268,7 @@ def test_analyze_trade_full(test_db_session, trade_with_alerts):
assert analysis["timing_score"] > 0 assert analysis["timing_score"] > 0
assert "prior_alerts" in analysis assert "prior_alerts" in analysis
assert len(analysis["prior_alerts"]) == 3 assert len(analysis["prior_alerts"]) == 3
# Check alert details # Check alert details
for alert_detail in analysis["prior_alerts"]: for alert_detail in analysis["prior_alerts"]:
assert "timestamp" in alert_detail assert "timestamp" in alert_detail
@@ -281,16 +282,15 @@ def test_analyze_recent_disclosures(test_db_session, trade_with_alerts, trade_wi
"""Test batch analysis of recent disclosures.""" """Test batch analysis of recent disclosures."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
# Both trades were created "recently" (in fixture setup) # Both trades were created "recently" (in fixture setup)
suspicious_trades = correlator.analyze_recent_disclosures( suspicious_trades = correlator.analyze_recent_disclosures(
days=365, # Wide window to catch test data days=365, min_timing_score=50 # Wide window to catch test data
min_timing_score=50
) )
# Should find at least the suspicious trade # Should find at least the suspicious trade
assert len(suspicious_trades) >= 1 assert len(suspicious_trades) >= 1
# Check sorting (highest score first) # Check sorting (highest score first)
if len(suspicious_trades) > 1: if len(suspicious_trades) > 1:
for i in range(len(suspicious_trades) - 1): for i in range(len(suspicious_trades) - 1):
@@ -301,12 +301,12 @@ def test_get_official_timing_pattern(test_db_session, trade_with_alerts):
"""Test official timing pattern analysis.""" """Test official timing pattern analysis."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
official = trade_with_alerts["official"] official = trade_with_alerts["official"]
# Use wide lookback to catch test data (trade is 2024-01-15) # Use wide lookback to catch test data (trade is 2024-01-15)
pattern = correlator.get_official_timing_pattern(official.id, lookback_days=3650) pattern = correlator.get_official_timing_pattern(official.id, lookback_days=3650)
assert pattern["official_id"] == official.id assert pattern["official_id"] == official.id
assert pattern["trade_count"] >= 1 assert pattern["trade_count"] >= 1
assert pattern["trades_with_prior_alerts"] >= 1 assert pattern["trades_with_prior_alerts"] >= 1
@@ -319,13 +319,13 @@ def test_get_official_timing_pattern_no_trades(test_db_session):
"""Test official with no trades.""" """Test official with no trades."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
official = Official(name="No Trades", chamber="House", party="Democrat", state="CA") official = Official(name="No Trades", chamber="House", party="Democrat", state="CA")
session.add(official) session.add(official)
session.commit() session.commit()
pattern = correlator.get_official_timing_pattern(official.id) pattern = correlator.get_official_timing_pattern(official.id)
assert pattern["trade_count"] == 0 assert pattern["trade_count"] == 0
assert "No trades" in pattern["pattern"] assert "No trades" in pattern["pattern"]
@@ -334,10 +334,10 @@ def test_get_ticker_timing_analysis(test_db_session, trade_with_alerts):
"""Test ticker timing analysis.""" """Test ticker timing analysis."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
# Use wide lookback to catch test data # Use wide lookback to catch test data
analysis = correlator.get_ticker_timing_analysis("NVDA", lookback_days=3650) analysis = correlator.get_ticker_timing_analysis("NVDA", lookback_days=3650)
assert analysis["ticker"] == "NVDA" assert analysis["ticker"] == "NVDA"
assert analysis["trade_count"] >= 1 assert analysis["trade_count"] >= 1
assert analysis["trades_with_alerts"] >= 1 assert analysis["trades_with_alerts"] >= 1
@@ -349,9 +349,9 @@ def test_get_ticker_timing_analysis_no_trades(test_db_session):
"""Test ticker with no trades.""" """Test ticker with no trades."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
analysis = correlator.get_ticker_timing_analysis("ZZZZ") analysis = correlator.get_ticker_timing_analysis("ZZZZ")
assert analysis["ticker"] == "ZZZZ" assert analysis["ticker"] == "ZZZZ"
assert analysis["trade_count"] == 0 assert analysis["trade_count"] == 0
assert "No trades" in analysis["pattern"] assert "No trades" in analysis["pattern"]
@@ -361,13 +361,13 @@ def test_alerts_outside_lookback_window(test_db_session):
"""Test that alerts outside lookback window are excluded.""" """Test that alerts outside lookback window are excluded."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
# Create trade and alerts # Create trade and alerts
official = Official(name="Test", chamber="House", party="Democrat", state="CA") official = Official(name="Test", chamber="House", party="Democrat", state="CA")
security = Security(ticker="TEST", name="Test Corp") security = Security(ticker="TEST", name="Test Corp")
session.add_all([official, security]) session.add_all([official, security])
session.flush() session.flush()
trade_date = date(2024, 1, 15) trade_date = date(2024, 1, 15)
trade = Trade( trade = Trade(
official_id=official.id, official_id=official.id,
@@ -379,29 +379,29 @@ def test_alerts_outside_lookback_window(test_db_session):
) )
session.add(trade) session.add(trade)
session.flush() session.flush()
# Alert 2 days before (within window) # Alert 2 days before (within window)
recent_alert = MarketAlert( recent_alert = MarketAlert(
ticker="TEST", ticker="TEST",
alert_type="test", alert_type="test",
timestamp=datetime(2024, 1, 13, 12, 0, tzinfo=timezone.utc), timestamp=datetime(2024, 1, 13, 12, 0, tzinfo=UTC),
severity=7, severity=7,
) )
# Alert 40 days before (outside 30-day window) # Alert 40 days before (outside 30-day window)
old_alert = MarketAlert( old_alert = MarketAlert(
ticker="TEST", ticker="TEST",
alert_type="test", alert_type="test",
timestamp=datetime(2023, 12, 6, 12, 0, tzinfo=timezone.utc), timestamp=datetime(2023, 12, 6, 12, 0, tzinfo=UTC),
severity=8, severity=8,
) )
session.add_all([recent_alert, old_alert]) session.add_all([recent_alert, old_alert])
session.commit() session.commit()
# Should only get recent alert # Should only get recent alert
alerts = correlator.get_alerts_before_trade(trade, lookback_days=30) alerts = correlator.get_alerts_before_trade(trade, lookback_days=30)
assert len(alerts) == 1 assert len(alerts) == 1
assert alerts[0].timestamp.date() == date(2024, 1, 13) assert alerts[0].timestamp.date() == date(2024, 1, 13)
@@ -410,14 +410,14 @@ def test_different_ticker_alerts_excluded(test_db_session):
"""Test that alerts for different tickers are excluded.""" """Test that alerts for different tickers are excluded."""
session = test_db_session session = test_db_session
correlator = DisclosureCorrelator(session) correlator = DisclosureCorrelator(session)
# Create trade for NVDA # Create trade for NVDA
official = Official(name="Test", chamber="House", party="Democrat", state="CA") official = Official(name="Test", chamber="House", party="Democrat", state="CA")
nvda = Security(ticker="NVDA", name="NVIDIA") nvda = Security(ticker="NVDA", name="NVIDIA")
msft = Security(ticker="MSFT", name="Microsoft") msft = Security(ticker="MSFT", name="Microsoft")
session.add_all([official, nvda, msft]) session.add_all([official, nvda, msft])
session.flush() session.flush()
trade = Trade( trade = Trade(
official_id=official.id, official_id=official.id,
security_id=nvda.id, security_id=nvda.id,
@@ -428,28 +428,27 @@ def test_different_ticker_alerts_excluded(test_db_session):
) )
session.add(trade) session.add(trade)
session.flush() session.flush()
# Create alerts for both tickers # Create alerts for both tickers
nvda_alert = MarketAlert( nvda_alert = MarketAlert(
ticker="NVDA", ticker="NVDA",
alert_type="test", alert_type="test",
timestamp=datetime(2024, 1, 10, 12, 0, tzinfo=timezone.utc), timestamp=datetime(2024, 1, 10, 12, 0, tzinfo=UTC),
severity=7, severity=7,
) )
msft_alert = MarketAlert( msft_alert = MarketAlert(
ticker="MSFT", ticker="MSFT",
alert_type="test", alert_type="test",
timestamp=datetime(2024, 1, 10, 12, 0, tzinfo=timezone.utc), timestamp=datetime(2024, 1, 10, 12, 0, tzinfo=UTC),
severity=8, severity=8,
) )
session.add_all([nvda_alert, msft_alert]) session.add_all([nvda_alert, msft_alert])
session.commit() session.commit()
# Should only get NVDA alert # Should only get NVDA alert
alerts = correlator.get_alerts_before_trade(trade, lookback_days=30) alerts = correlator.get_alerts_before_trade(trade, lookback_days=30)
assert len(alerts) == 1 assert len(alerts) == 1
assert alerts[0].ticker == "NVDA" assert alerts[0].ticker == "NVDA"
+5 -10
View File
@@ -5,6 +5,7 @@ Tests for database models.
from datetime import date from datetime import date
from decimal import Decimal from decimal import Decimal
import pytest
from sqlalchemy import select from sqlalchemy import select
from pote.db.models import Price, Security, Trade from pote.db.models import Price, Security, Trade
@@ -56,12 +57,9 @@ def test_unique_constraints(test_db_session, sample_security):
dup_security = Security(ticker="AAPL", name="Apple Duplicate") dup_security = Security(ticker="AAPL", name="Apple Duplicate")
test_db_session.add(dup_security) test_db_session.add(dup_security)
try: with pytest.raises(IntegrityError):
test_db_session.commit() test_db_session.commit()
assert False, "Should have raised IntegrityError" test_db_session.rollback()
except IntegrityError:
test_db_session.rollback()
# Expected behavior
def test_price_unique_per_security_date(test_db_session, sample_security): def test_price_unique_per_security_date(test_db_session, sample_security):
@@ -83,12 +81,9 @@ def test_price_unique_per_security_date(test_db_session, sample_security):
) )
test_db_session.add(price2) test_db_session.add(price2)
try: with pytest.raises(IntegrityError):
test_db_session.commit() test_db_session.commit()
assert False, "Should have raised IntegrityError" test_db_session.rollback()
except IntegrityError:
test_db_session.rollback()
# Expected behavior
def test_trade_queries(test_db_session, sample_official, sample_security): def test_trade_queries(test_db_session, sample_official, sample_security):
+113 -83
View File
@@ -1,25 +1,26 @@
"""Tests for market monitoring module.""" """Tests for market monitoring module."""
import pytest from datetime import UTC, date, datetime, timedelta
from datetime import date, datetime, timedelta, timezone
from decimal import Decimal from decimal import Decimal
from pote.monitoring.market_monitor import MarketMonitor import pytest
from pote.db.models import MarketAlert, Official, Security, Trade
from pote.monitoring.alert_manager import AlertManager from pote.monitoring.alert_manager import AlertManager
from pote.db.models import Official, Security, Trade, MarketAlert from pote.monitoring.market_monitor import MarketMonitor
@pytest.fixture @pytest.fixture
def sample_congressional_trades(test_db_session): def sample_congressional_trades(test_db_session):
"""Create sample congressional trades for watchlist building.""" """Create sample congressional trades for watchlist building."""
session = test_db_session session = test_db_session
# Create officials # Create officials
pelosi = Official(name="Nancy Pelosi", chamber="House", party="Democrat", state="CA") pelosi = Official(name="Nancy Pelosi", chamber="House", party="Democrat", state="CA")
tuberville = Official(name="Tommy Tuberville", chamber="Senate", party="Republican", state="AL") tuberville = Official(name="Tommy Tuberville", chamber="Senate", party="Republican", state="AL")
session.add_all([pelosi, tuberville]) session.add_all([pelosi, tuberville])
session.flush() session.flush()
# Create securities # Create securities
nvda = Security(ticker="NVDA", name="NVIDIA Corporation", sector="Technology") nvda = Security(ticker="NVDA", name="NVIDIA Corporation", sector="Technology")
msft = Security(ticker="MSFT", name="Microsoft Corporation", sector="Technology") msft = Security(ticker="MSFT", name="Microsoft Corporation", sector="Technology")
@@ -28,28 +29,58 @@ def sample_congressional_trades(test_db_session):
spy = Security(ticker="SPY", name="SPDR S&P 500 ETF", sector="Financial") spy = Security(ticker="SPY", name="SPDR S&P 500 ETF", sector="Financial")
session.add_all([nvda, msft, aapl, tsla, spy]) session.add_all([nvda, msft, aapl, tsla, spy])
session.flush() session.flush()
# Create multiple trades (NVDA is most traded) # Create multiple trades (NVDA is most traded)
trades = [ trades = [
Trade(official_id=pelosi.id, security_id=nvda.id, source="test", Trade(
transaction_date=date(2024, 1, 15), side="buy", official_id=pelosi.id,
value_min=Decimal("15001"), value_max=Decimal("50000")), security_id=nvda.id,
Trade(official_id=pelosi.id, security_id=nvda.id, source="test", source="test",
transaction_date=date(2024, 2, 1), side="buy", transaction_date=date(2024, 1, 15),
value_min=Decimal("15001"), value_max=Decimal("50000")), side="buy",
Trade(official_id=tuberville.id, security_id=nvda.id, source="test", value_min=Decimal("15001"),
transaction_date=date(2024, 2, 15), side="buy", value_max=Decimal("50000"),
value_min=Decimal("50001"), value_max=Decimal("100000")), ),
Trade(official_id=pelosi.id, security_id=msft.id, source="test", Trade(
transaction_date=date(2024, 1, 20), side="sell", official_id=pelosi.id,
value_min=Decimal("15001"), value_max=Decimal("50000")), security_id=nvda.id,
Trade(official_id=tuberville.id, security_id=aapl.id, source="test", source="test",
transaction_date=date(2024, 2, 10), side="buy", transaction_date=date(2024, 2, 1),
value_min=Decimal("15001"), value_max=Decimal("50000")), side="buy",
value_min=Decimal("15001"),
value_max=Decimal("50000"),
),
Trade(
official_id=tuberville.id,
security_id=nvda.id,
source="test",
transaction_date=date(2024, 2, 15),
side="buy",
value_min=Decimal("50001"),
value_max=Decimal("100000"),
),
Trade(
official_id=pelosi.id,
security_id=msft.id,
source="test",
transaction_date=date(2024, 1, 20),
side="sell",
value_min=Decimal("15001"),
value_max=Decimal("50000"),
),
Trade(
official_id=tuberville.id,
security_id=aapl.id,
source="test",
transaction_date=date(2024, 2, 10),
side="buy",
value_min=Decimal("15001"),
value_max=Decimal("50000"),
),
] ]
session.add_all(trades) session.add_all(trades)
session.commit() session.commit()
return { return {
"officials": [pelosi, tuberville], "officials": [pelosi, tuberville],
"securities": [nvda, msft, aapl, tsla, spy], "securities": [nvda, msft, aapl, tsla, spy],
@@ -61,9 +92,9 @@ def sample_congressional_trades(test_db_session):
def sample_alerts(test_db_session): def sample_alerts(test_db_session):
"""Create sample market alerts.""" """Create sample market alerts."""
session = test_db_session session = test_db_session
now = datetime.now(timezone.utc) now = datetime.now(UTC)
alerts = [ alerts = [
MarketAlert( MarketAlert(
ticker="NVDA", ticker="NVDA",
@@ -96,10 +127,10 @@ def sample_alerts(test_db_session):
severity=5, severity=5,
), ),
] ]
session.add_all(alerts) session.add_all(alerts)
session.commit() session.commit()
return alerts return alerts
@@ -107,9 +138,9 @@ def test_get_congressional_watchlist(test_db_session, sample_congressional_trade
"""Test building watchlist from congressional trades.""" """Test building watchlist from congressional trades."""
session = test_db_session session = test_db_session
monitor = MarketMonitor(session) monitor = MarketMonitor(session)
watchlist = monitor.get_congressional_watchlist(limit=10) watchlist = monitor.get_congressional_watchlist(limit=10)
assert len(watchlist) > 0 assert len(watchlist) > 0
assert "NVDA" in watchlist # Most traded assert "NVDA" in watchlist # Most traded
assert watchlist[0] == "NVDA" # Should be first (3 trades) assert watchlist[0] == "NVDA" # Should be first (3 trades)
@@ -119,11 +150,11 @@ def test_check_ticker_basic(test_db_session):
"""Test basic ticker checking (may not find alerts with real data).""" """Test basic ticker checking (may not find alerts with real data)."""
session = test_db_session session = test_db_session
monitor = MarketMonitor(session) monitor = MarketMonitor(session)
# This uses real yfinance data, so alerts depend on current market # This uses real yfinance data, so alerts depend on current market
# We test that it doesn't crash # We test that it doesn't crash
alerts = monitor.check_ticker("AAPL", lookback_days=5) alerts = monitor.check_ticker("AAPL", lookback_days=5)
assert isinstance(alerts, list) assert isinstance(alerts, list)
# Each alert should have required fields # Each alert should have required fields
for alert in alerts: for alert in alerts:
@@ -137,7 +168,7 @@ def test_scan_watchlist_with_mock(test_db_session, sample_congressional_trades,
"""Test scanning watchlist with mocked data.""" """Test scanning watchlist with mocked data."""
session = test_db_session session = test_db_session
monitor = MarketMonitor(session) monitor = MarketMonitor(session)
# Mock the check_ticker method to return controlled data # Mock the check_ticker method to return controlled data
def mock_check_ticker(ticker, lookback_days=5): def mock_check_ticker(ticker, lookback_days=5):
if ticker == "NVDA": if ticker == "NVDA":
@@ -145,7 +176,7 @@ def test_scan_watchlist_with_mock(test_db_session, sample_congressional_trades,
{ {
"ticker": ticker, "ticker": ticker,
"alert_type": "unusual_volume", "alert_type": "unusual_volume",
"timestamp": datetime.now(timezone.utc), "timestamp": datetime.now(UTC),
"details": {"multiplier": 3.5}, "details": {"multiplier": 3.5},
"price": Decimal("500.00"), "price": Decimal("500.00"),
"volume": 100000000, "volume": 100000000,
@@ -154,12 +185,12 @@ def test_scan_watchlist_with_mock(test_db_session, sample_congressional_trades,
} }
] ]
return [] return []
monkeypatch.setattr(monitor, "check_ticker", mock_check_ticker) monkeypatch.setattr(monitor, "check_ticker", mock_check_ticker)
# Scan with limited watchlist # Scan with limited watchlist
alerts = monitor.scan_watchlist(tickers=["NVDA", "MSFT"], lookback_days=5) alerts = monitor.scan_watchlist(tickers=["NVDA", "MSFT"], lookback_days=5)
assert len(alerts) == 1 assert len(alerts) == 1
assert alerts[0]["ticker"] == "NVDA" assert alerts[0]["ticker"] == "NVDA"
assert alerts[0]["alert_type"] == "unusual_volume" assert alerts[0]["alert_type"] == "unusual_volume"
@@ -169,12 +200,12 @@ def test_save_alerts(test_db_session):
"""Test saving alerts to database.""" """Test saving alerts to database."""
session = test_db_session session = test_db_session
monitor = MarketMonitor(session) monitor = MarketMonitor(session)
alerts_data = [ alerts_data = [
{ {
"ticker": "TSLA", "ticker": "TSLA",
"alert_type": "price_spike", "alert_type": "price_spike",
"timestamp": datetime.now(timezone.utc), "timestamp": datetime.now(UTC),
"details": {"change_pct": 7.5}, "details": {"change_pct": 7.5},
"price": Decimal("250.00"), "price": Decimal("250.00"),
"volume": 75000000, "volume": 75000000,
@@ -184,7 +215,7 @@ def test_save_alerts(test_db_session):
{ {
"ticker": "TSLA", "ticker": "TSLA",
"alert_type": "unusual_volume", "alert_type": "unusual_volume",
"timestamp": datetime.now(timezone.utc), "timestamp": datetime.now(UTC),
"details": {"multiplier": 4.0}, "details": {"multiplier": 4.0},
"price": Decimal("250.00"), "price": Decimal("250.00"),
"volume": 120000000, "volume": 120000000,
@@ -192,11 +223,11 @@ def test_save_alerts(test_db_session):
"severity": 9, "severity": 9,
}, },
] ]
saved_count = monitor.save_alerts(alerts_data) saved_count = monitor.save_alerts(alerts_data)
assert saved_count == 2 assert saved_count == 2
# Verify in database # Verify in database
alerts = session.query(MarketAlert).filter_by(ticker="TSLA").all() alerts = session.query(MarketAlert).filter_by(ticker="TSLA").all()
assert len(alerts) == 2 assert len(alerts) == 2
@@ -206,21 +237,21 @@ def test_get_recent_alerts(test_db_session, sample_alerts):
"""Test querying recent alerts.""" """Test querying recent alerts."""
session = test_db_session session = test_db_session
monitor = MarketMonitor(session) monitor = MarketMonitor(session)
# Get all alerts # Get all alerts
all_alerts = monitor.get_recent_alerts(days=1) all_alerts = monitor.get_recent_alerts(days=1)
assert len(all_alerts) >= 3 assert len(all_alerts) >= 3
# Filter by ticker # Filter by ticker
nvda_alerts = monitor.get_recent_alerts(ticker="NVDA", days=1) nvda_alerts = monitor.get_recent_alerts(ticker="NVDA", days=1)
assert len(nvda_alerts) == 2 assert len(nvda_alerts) == 2
assert all(a.ticker == "NVDA" for a in nvda_alerts) assert all(a.ticker == "NVDA" for a in nvda_alerts)
# Filter by alert type # Filter by alert type
volume_alerts = monitor.get_recent_alerts(alert_type="unusual_volume", days=1) volume_alerts = monitor.get_recent_alerts(alert_type="unusual_volume", days=1)
assert len(volume_alerts) == 1 assert len(volume_alerts) == 1
assert volume_alerts[0].alert_type == "unusual_volume" assert volume_alerts[0].alert_type == "unusual_volume"
# Filter by severity # Filter by severity
high_sev_alerts = monitor.get_recent_alerts(min_severity=6, days=1) high_sev_alerts = monitor.get_recent_alerts(min_severity=6, days=1)
assert all(a.severity >= 6 for a in high_sev_alerts) assert all(a.severity >= 6 for a in high_sev_alerts)
@@ -230,12 +261,12 @@ def test_get_ticker_alert_summary(test_db_session, sample_alerts):
"""Test alert summary by ticker.""" """Test alert summary by ticker."""
session = test_db_session session = test_db_session
monitor = MarketMonitor(session) monitor = MarketMonitor(session)
summary = monitor.get_ticker_alert_summary(days=1) summary = monitor.get_ticker_alert_summary(days=1)
assert "NVDA" in summary assert "NVDA" in summary
assert "MSFT" in summary assert "MSFT" in summary
nvda_summary = summary["NVDA"] nvda_summary = summary["NVDA"]
assert nvda_summary["alert_count"] == 2 assert nvda_summary["alert_count"] == 2
assert nvda_summary["max_severity"] == 7 assert nvda_summary["max_severity"] == 7
@@ -246,11 +277,11 @@ def test_alert_manager_format_text(test_db_session, sample_alerts):
"""Test text formatting of alerts.""" """Test text formatting of alerts."""
session = test_db_session session = test_db_session
alert_mgr = AlertManager(session) alert_mgr = AlertManager(session)
alert = sample_alerts[0] # NVDA unusual volume alert = sample_alerts[0] # NVDA unusual volume
text = alert_mgr.format_alert_text(alert) text = alert_mgr.format_alert_text(alert)
assert "NVDA" in text assert "NVDA" in text
assert "UNUSUAL VOLUME" in text assert "UNUSUAL VOLUME" in text
assert "Severity" in text assert "Severity" in text
@@ -261,11 +292,11 @@ def test_alert_manager_format_html(test_db_session, sample_alerts):
"""Test HTML formatting of alerts.""" """Test HTML formatting of alerts."""
session = test_db_session session = test_db_session
alert_mgr = AlertManager(session) alert_mgr = AlertManager(session)
alert = sample_alerts[0] alert = sample_alerts[0]
html = alert_mgr.format_alert_html(alert) html = alert_mgr.format_alert_html(alert)
assert "<div" in html assert "<div" in html
assert "NVDA" in html assert "NVDA" in html
assert "unusual_volume" in html or "Unusual Volume" in html assert "unusual_volume" in html or "Unusual Volume" in html
@@ -275,29 +306,29 @@ def test_alert_manager_filter_alerts(test_db_session, sample_alerts):
"""Test filtering alerts.""" """Test filtering alerts."""
session = test_db_session session = test_db_session
alert_mgr = AlertManager(session) alert_mgr = AlertManager(session)
# Filter by severity # Filter by severity
high_sev = alert_mgr.filter_alerts(sample_alerts, min_severity=6) high_sev = alert_mgr.filter_alerts(sample_alerts, min_severity=6)
assert len(high_sev) == 1 assert len(high_sev) == 1
assert high_sev[0].ticker == "NVDA" assert high_sev[0].ticker == "NVDA"
assert high_sev[0].severity == 7 assert high_sev[0].severity == 7
# Filter by ticker # Filter by ticker
nvda_only = alert_mgr.filter_alerts(sample_alerts, min_severity=0, tickers=["NVDA"]) nvda_only = alert_mgr.filter_alerts(sample_alerts, min_severity=0, tickers=["NVDA"])
assert len(nvda_only) == 2 assert len(nvda_only) == 2
assert all(a.ticker == "NVDA" for a in nvda_only) assert all(a.ticker == "NVDA" for a in nvda_only)
# Filter by alert type # Filter by alert type
volume_only = alert_mgr.filter_alerts(sample_alerts, alert_types=["unusual_volume"]) volume_only = alert_mgr.filter_alerts(sample_alerts, alert_types=["unusual_volume"])
assert len(volume_only) == 1 assert len(volume_only) == 1
assert volume_only[0].alert_type == "unusual_volume" assert volume_only[0].alert_type == "unusual_volume"
# Combined filters # Combined filters
filtered = alert_mgr.filter_alerts( filtered = alert_mgr.filter_alerts(
sample_alerts, sample_alerts,
min_severity=4, min_severity=4,
tickers=["NVDA"], tickers=["NVDA"],
alert_types=["unusual_volume", "price_spike"] alert_types=["unusual_volume", "price_spike"],
) )
assert len(filtered) == 2 assert len(filtered) == 2
@@ -306,9 +337,9 @@ def test_alert_manager_generate_summary_text(test_db_session, sample_alerts):
"""Test generating text summary report.""" """Test generating text summary report."""
session = test_db_session session = test_db_session
alert_mgr = AlertManager(session) alert_mgr = AlertManager(session)
report = alert_mgr.generate_summary_report(sample_alerts, format="text") report = alert_mgr.generate_summary_report(sample_alerts, output_format="text")
assert "MARKET ACTIVITY ALERTS" in report assert "MARKET ACTIVITY ALERTS" in report
assert "3 Alerts" in report assert "3 Alerts" in report
assert "NVDA" in report assert "NVDA" in report
@@ -320,9 +351,9 @@ def test_alert_manager_generate_summary_html(test_db_session, sample_alerts):
"""Test generating HTML summary report.""" """Test generating HTML summary report."""
session = test_db_session session = test_db_session
alert_mgr = AlertManager(session) alert_mgr = AlertManager(session)
report = alert_mgr.generate_summary_report(sample_alerts, format="html") report = alert_mgr.generate_summary_report(sample_alerts, output_format="html")
assert "<html>" in report assert "<html>" in report
assert "<head>" in report assert "<head>" in report
assert "Market Activity Alerts" in report assert "Market Activity Alerts" in report
@@ -333,20 +364,20 @@ def test_alert_manager_empty_alerts(test_db_session):
"""Test handling empty alert list.""" """Test handling empty alert list."""
session = test_db_session session = test_db_session
alert_mgr = AlertManager(session) alert_mgr = AlertManager(session)
report = alert_mgr.generate_summary_report([], format="text") report = alert_mgr.generate_summary_report([], output_format="text")
assert "No alerts" in report assert "No alerts" in report
def test_market_alert_model(test_db_session): def test_market_alert_model(test_db_session):
"""Test MarketAlert model creation and retrieval.""" """Test MarketAlert model creation and retrieval."""
session = test_db_session session = test_db_session
alert = MarketAlert( alert = MarketAlert(
ticker="GOOGL", ticker="GOOGL",
alert_type="price_spike", alert_type="price_spike",
timestamp=datetime.now(timezone.utc), timestamp=datetime.now(UTC),
details={"test": "data"}, details={"test": "data"},
price=Decimal("140.50"), price=Decimal("140.50"),
volume=25000000, volume=25000000,
@@ -354,13 +385,13 @@ def test_market_alert_model(test_db_session):
severity=7, severity=7,
source="test", source="test",
) )
session.add(alert) session.add(alert)
session.commit() session.commit()
# Retrieve # Retrieve
retrieved = session.query(MarketAlert).filter_by(ticker="GOOGL").first() retrieved = session.query(MarketAlert).filter_by(ticker="GOOGL").first()
assert retrieved is not None assert retrieved is not None
assert retrieved.ticker == "GOOGL" assert retrieved.ticker == "GOOGL"
assert retrieved.alert_type == "price_spike" assert retrieved.alert_type == "price_spike"
@@ -372,9 +403,9 @@ def test_market_alert_model(test_db_session):
def test_alert_timestamp_filtering(test_db_session): def test_alert_timestamp_filtering(test_db_session):
"""Test filtering alerts by timestamp.""" """Test filtering alerts by timestamp."""
session = test_db_session session = test_db_session
now = datetime.now(timezone.utc) now = datetime.now(UTC)
# Create alerts at different times # Create alerts at different times
old_alert = MarketAlert( old_alert = MarketAlert(
ticker="TEST1", ticker="TEST1",
@@ -388,20 +419,19 @@ def test_alert_timestamp_filtering(test_db_session):
timestamp=now - timedelta(hours=2), timestamp=now - timedelta(hours=2),
severity=5, severity=5,
) )
session.add_all([old_alert, recent_alert]) session.add_all([old_alert, recent_alert])
session.commit() session.commit()
monitor = MarketMonitor(session) monitor = MarketMonitor(session)
# Should only get recent alert # Should only get recent alert
alerts_1_day = monitor.get_recent_alerts(days=1) alerts_1_day = monitor.get_recent_alerts(days=1)
test_alerts = [a for a in alerts_1_day if a.ticker.startswith("TEST")] test_alerts = [a for a in alerts_1_day if a.ticker.startswith("TEST")]
assert len(test_alerts) == 1 assert len(test_alerts) == 1
assert test_alerts[0].ticker == "TEST2" assert test_alerts[0].ticker == "TEST2"
# Should get both with longer lookback # Should get both with longer lookback
alerts_30_days = monitor.get_recent_alerts(days=30) alerts_30_days = monitor.get_recent_alerts(days=30)
test_alerts = [a for a in alerts_30_days if a.ticker.startswith("TEST")] test_alerts = [a for a in alerts_30_days if a.ticker.startswith("TEST")]
assert len(test_alerts) == 2 assert len(test_alerts) == 2
+64 -71
View File
@@ -1,38 +1,39 @@
"""Tests for pattern detection module.""" """Tests for pattern detection module."""
import pytest from datetime import UTC, date, datetime, timedelta
from datetime import date, datetime, timedelta, timezone
from decimal import Decimal from decimal import Decimal
import pytest
from pote.db.models import MarketAlert, Official, Security, Trade
from pote.monitoring.pattern_detector import PatternDetector from pote.monitoring.pattern_detector import PatternDetector
from pote.db.models import Official, Security, Trade, MarketAlert
@pytest.fixture @pytest.fixture
def multiple_officials_with_patterns(test_db_session): def multiple_officials_with_patterns(test_db_session):
"""Create multiple officials with different timing patterns.""" """Create multiple officials with different timing patterns."""
session = test_db_session session = test_db_session
# Create officials # Create officials
pelosi = Official(name="Nancy Pelosi", chamber="House", party="Democrat", state="CA") pelosi = Official(name="Nancy Pelosi", chamber="House", party="Democrat", state="CA")
tuberville = Official(name="Tommy Tuberville", chamber="Senate", party="Republican", state="AL") tuberville = Official(name="Tommy Tuberville", chamber="Senate", party="Republican", state="AL")
clean_trader = Official(name="Clean Trader", chamber="House", party="Independent", state="TX") clean_trader = Official(name="Clean Trader", chamber="House", party="Independent", state="TX")
session.add_all([pelosi, tuberville, clean_trader]) session.add_all([pelosi, tuberville, clean_trader])
session.flush() session.flush()
# Create securities # Create securities
nvda = Security(ticker="NVDA", name="NVIDIA", sector="Technology") nvda = Security(ticker="NVDA", name="NVIDIA", sector="Technology")
msft = Security(ticker="MSFT", name="Microsoft", sector="Technology") msft = Security(ticker="MSFT", name="Microsoft", sector="Technology")
xom = Security(ticker="XOM", name="Exxon", sector="Energy") xom = Security(ticker="XOM", name="Exxon", sector="Energy")
session.add_all([nvda, msft, xom]) session.add_all([nvda, msft, xom])
session.flush() session.flush()
# Pelosi - Suspicious pattern (trades with alerts) # Pelosi - Suspicious pattern (trades with alerts)
for i in range(5): for i in range(5):
trade_date = date(2024, 1, 15) + timedelta(days=i*30) trade_date = date(2024, 1, 15) + timedelta(days=i * 30)
# Create trade # Create trade
trade = Trade( trade = Trade(
official_id=pelosi.id, official_id=pelosi.id,
@@ -45,24 +46,23 @@ def multiple_officials_with_patterns(test_db_session):
) )
session.add(trade) session.add(trade)
session.flush() session.flush()
# Create alerts BEFORE trade (suspicious) # Create alerts BEFORE trade (suspicious)
for j in range(2): for j in range(2):
alert = MarketAlert( alert = MarketAlert(
ticker="NVDA", ticker="NVDA",
alert_type="unusual_volume", alert_type="unusual_volume",
timestamp=datetime.combine( timestamp=datetime.combine(
trade_date - timedelta(days=3+j), trade_date - timedelta(days=3 + j), datetime.min.time()
datetime.min.time() ).replace(tzinfo=UTC),
).replace(tzinfo=timezone.utc),
severity=7 + j, severity=7 + j,
) )
session.add(alert) session.add(alert)
# Tuberville - Mixed pattern # Tuberville - Mixed pattern
for i in range(4): for i in range(4):
trade_date = date(2024, 2, 1) + timedelta(days=i*30) trade_date = date(2024, 2, 1) + timedelta(days=i * 30)
trade = Trade( trade = Trade(
official_id=tuberville.id, official_id=tuberville.id,
security_id=msft.id, security_id=msft.id,
@@ -74,24 +74,23 @@ def multiple_officials_with_patterns(test_db_session):
) )
session.add(trade) session.add(trade)
session.flush() session.flush()
# Only first 2 trades have alerts # Only first 2 trades have alerts
if i < 2: if i < 2:
alert = MarketAlert( alert = MarketAlert(
ticker="MSFT", ticker="MSFT",
alert_type="price_spike", alert_type="price_spike",
timestamp=datetime.combine( timestamp=datetime.combine(
trade_date - timedelta(days=5), trade_date - timedelta(days=5), datetime.min.time()
datetime.min.time() ).replace(tzinfo=UTC),
).replace(tzinfo=timezone.utc),
severity=6, severity=6,
) )
session.add(alert) session.add(alert)
# Clean trader - No suspicious activity # Clean trader - No suspicious activity
for i in range(3): for i in range(3):
trade_date = date(2024, 3, 1) + timedelta(days=i*30) trade_date = date(2024, 3, 1) + timedelta(days=i * 30)
trade = Trade( trade = Trade(
official_id=clean_trader.id, official_id=clean_trader.id,
security_id=xom.id, security_id=xom.id,
@@ -102,9 +101,9 @@ def multiple_officials_with_patterns(test_db_session):
value_max=Decimal("50000"), value_max=Decimal("50000"),
) )
session.add(trade) session.add(trade)
session.commit() session.commit()
return { return {
"officials": [pelosi, tuberville, clean_trader], "officials": [pelosi, tuberville, clean_trader],
"securities": [nvda, msft, xom], "securities": [nvda, msft, xom],
@@ -115,15 +114,15 @@ def test_rank_officials_by_timing(test_db_session, multiple_officials_with_patte
"""Test ranking officials by timing scores.""" """Test ranking officials by timing scores."""
session = test_db_session session = test_db_session
detector = PatternDetector(session) detector = PatternDetector(session)
rankings = detector.rank_officials_by_timing(lookback_days=3650, min_trades=3) rankings = detector.rank_officials_by_timing(lookback_days=3650, min_trades=3)
assert len(rankings) >= 2 # At least 2 officials with 3+ trades assert len(rankings) >= 2 # At least 2 officials with 3+ trades
# Rankings should be sorted by avg_timing_score (descending) # Rankings should be sorted by avg_timing_score (descending)
for i in range(len(rankings) - 1): for i in range(len(rankings) - 1):
assert rankings[i]["avg_timing_score"] >= rankings[i + 1]["avg_timing_score"] assert rankings[i]["avg_timing_score"] >= rankings[i + 1]["avg_timing_score"]
# Check required fields # Check required fields
for ranking in rankings: for ranking in rankings:
assert "name" in ranking assert "name" in ranking
@@ -138,16 +137,15 @@ def test_identify_repeat_offenders(test_db_session, multiple_officials_with_patt
"""Test identifying repeat offenders.""" """Test identifying repeat offenders."""
session = test_db_session session = test_db_session
detector = PatternDetector(session) detector = PatternDetector(session)
# Set low threshold to catch Pelosi (who has 100% suspicious rate) # Set low threshold to catch Pelosi (who has 100% suspicious rate)
offenders = detector.identify_repeat_offenders( offenders = detector.identify_repeat_offenders(
lookback_days=3650, lookback_days=3650, min_suspicious_rate=0.7 # 70%+
min_suspicious_rate=0.7 # 70%+
) )
# Should find at least Pelosi (all trades with alerts) # Should find at least Pelosi (all trades with alerts)
assert isinstance(offenders, list) assert isinstance(offenders, list)
# All offenders should have high suspicious rates # All offenders should have high suspicious rates
for offender in offenders: for offender in offenders:
assert offender["suspicious_rate"] >= 70 assert offender["suspicious_rate"] >= 70
@@ -157,19 +155,16 @@ def test_analyze_ticker_patterns(test_db_session, multiple_officials_with_patter
"""Test ticker pattern analysis.""" """Test ticker pattern analysis."""
session = test_db_session session = test_db_session
detector = PatternDetector(session) detector = PatternDetector(session)
ticker_patterns = detector.analyze_ticker_patterns( ticker_patterns = detector.analyze_ticker_patterns(lookback_days=3650, min_trades=3)
lookback_days=3650,
min_trades=3
)
assert isinstance(ticker_patterns, list) assert isinstance(ticker_patterns, list)
assert len(ticker_patterns) >= 1 # At least NVDA should qualify assert len(ticker_patterns) >= 1 # At least NVDA should qualify
# Check sorting # Check sorting
for i in range(len(ticker_patterns) - 1): for i in range(len(ticker_patterns) - 1):
assert ticker_patterns[i]["avg_timing_score"] >= ticker_patterns[i + 1]["avg_timing_score"] assert ticker_patterns[i]["avg_timing_score"] >= ticker_patterns[i + 1]["avg_timing_score"]
# Check fields # Check fields
for pattern in ticker_patterns: for pattern in ticker_patterns:
assert "ticker" in pattern assert "ticker" in pattern
@@ -182,12 +177,12 @@ def test_get_sector_timing_analysis(test_db_session, multiple_officials_with_pat
"""Test sector timing analysis.""" """Test sector timing analysis."""
session = test_db_session session = test_db_session
detector = PatternDetector(session) detector = PatternDetector(session)
sector_stats = detector.get_sector_timing_analysis(lookback_days=3650) sector_stats = detector.get_sector_timing_analysis(lookback_days=3650)
assert isinstance(sector_stats, dict) assert isinstance(sector_stats, dict)
assert len(sector_stats) >= 2 # Technology and Energy assert len(sector_stats) >= 2 # Technology and Energy
# Check Technology sector (should have alerts) # Check Technology sector (should have alerts)
if "Technology" in sector_stats: if "Technology" in sector_stats:
tech = sector_stats["Technology"] tech = sector_stats["Technology"]
@@ -201,14 +196,14 @@ def test_get_party_comparison(test_db_session, multiple_officials_with_patterns)
"""Test party comparison analysis.""" """Test party comparison analysis."""
session = test_db_session session = test_db_session
detector = PatternDetector(session) detector = PatternDetector(session)
party_stats = detector.get_party_comparison(lookback_days=3650) party_stats = detector.get_party_comparison(lookback_days=3650)
assert isinstance(party_stats, dict) assert isinstance(party_stats, dict)
assert len(party_stats) >= 2 # Democrat, Republican, Independent assert len(party_stats) >= 2 # Democrat, Republican, Independent
# Check that we have data for each party # Check that we have data for each party
for party, stats in party_stats.items(): for stats in party_stats.values():
assert "official_count" in stats assert "official_count" in stats
assert "total_trades" in stats assert "total_trades" in stats
assert "avg_timing_score" in stats assert "avg_timing_score" in stats
@@ -219,9 +214,9 @@ def test_generate_pattern_report(test_db_session, multiple_officials_with_patter
"""Test comprehensive pattern report generation.""" """Test comprehensive pattern report generation."""
session = test_db_session session = test_db_session
detector = PatternDetector(session) detector = PatternDetector(session)
report = detector.generate_pattern_report(lookback_days=3650) report = detector.generate_pattern_report(lookback_days=3650)
# Check report structure # Check report structure
assert "period_days" in report assert "period_days" in report
assert "summary" in report assert "summary" in report
@@ -230,12 +225,12 @@ def test_generate_pattern_report(test_db_session, multiple_officials_with_patter
assert "suspicious_tickers" in report assert "suspicious_tickers" in report
assert "sector_analysis" in report assert "sector_analysis" in report
assert "party_comparison" in report assert "party_comparison" in report
# Check summary # Check summary
summary = report["summary"] summary = report["summary"]
assert summary["total_officials_analyzed"] >= 2 assert summary["total_officials_analyzed"] >= 2
assert "avg_timing_score" in summary assert "avg_timing_score" in summary
# Check that lists are populated # Check that lists are populated
assert len(report["top_suspicious_officials"]) >= 2 assert len(report["top_suspicious_officials"]) >= 2
assert isinstance(report["suspicious_tickers"], list) assert isinstance(report["suspicious_tickers"], list)
@@ -245,15 +240,15 @@ def test_rank_officials_min_trades_filter(test_db_session, multiple_officials_wi
"""Test that min_trades filter works correctly.""" """Test that min_trades filter works correctly."""
session = test_db_session session = test_db_session
detector = PatternDetector(session) detector = PatternDetector(session)
# With min_trades=5, should only get Pelosi # With min_trades=5, should only get Pelosi
rankings_high = detector.rank_officials_by_timing(lookback_days=3650, min_trades=5) rankings_high = detector.rank_officials_by_timing(lookback_days=3650, min_trades=5)
# With min_trades=3, should get at least 2 officials # With min_trades=3, should get at least 2 officials
rankings_low = detector.rank_officials_by_timing(lookback_days=3650, min_trades=3) rankings_low = detector.rank_officials_by_timing(lookback_days=3650, min_trades=3)
assert len(rankings_low) >= len(rankings_high) assert len(rankings_low) >= len(rankings_high)
# All officials should meet min_trades requirement # All officials should meet min_trades requirement
for ranking in rankings_high: for ranking in rankings_high:
assert ranking["trade_count"] >= 5 assert ranking["trade_count"] >= 5
@@ -263,17 +258,17 @@ def test_empty_data_handling(test_db_session):
"""Test handling of empty dataset.""" """Test handling of empty dataset."""
session = test_db_session session = test_db_session
detector = PatternDetector(session) detector = PatternDetector(session)
# With no data, should return empty results # With no data, should return empty results
rankings = detector.rank_officials_by_timing(lookback_days=30, min_trades=1) rankings = detector.rank_officials_by_timing(lookback_days=30, min_trades=1)
assert rankings == [] assert rankings == []
offenders = detector.identify_repeat_offenders(lookback_days=30) offenders = detector.identify_repeat_offenders(lookback_days=30)
assert offenders == [] assert offenders == []
tickers = detector.analyze_ticker_patterns(lookback_days=30) tickers = detector.analyze_ticker_patterns(lookback_days=30)
assert tickers == [] assert tickers == []
sectors = detector.get_sector_timing_analysis(lookback_days=30) sectors = detector.get_sector_timing_analysis(lookback_days=30)
assert sectors == {} assert sectors == {}
@@ -282,13 +277,13 @@ def test_ranking_score_accuracy(test_db_session, multiple_officials_with_pattern
"""Test that rankings accurately reflect timing patterns.""" """Test that rankings accurately reflect timing patterns."""
session = test_db_session session = test_db_session
detector = PatternDetector(session) detector = PatternDetector(session)
rankings = detector.rank_officials_by_timing(lookback_days=3650, min_trades=3) rankings = detector.rank_officials_by_timing(lookback_days=3650, min_trades=3)
# Find Pelosi and Clean Trader # Find Pelosi and Clean Trader
pelosi_rank = next((r for r in rankings if "Pelosi" in r["name"]), None) pelosi_rank = next((r for r in rankings if "Pelosi" in r["name"]), None)
clean_rank = next((r for r in rankings if "Clean" in r["name"]), None) clean_rank = next((r for r in rankings if "Clean" in r["name"]), None)
if pelosi_rank and clean_rank: if pelosi_rank and clean_rank:
# Pelosi (with alerts) should have higher score than clean trader (no alerts) # Pelosi (with alerts) should have higher score than clean trader (no alerts)
assert pelosi_rank["avg_timing_score"] > clean_rank["avg_timing_score"] assert pelosi_rank["avg_timing_score"] > clean_rank["avg_timing_score"]
@@ -299,9 +294,9 @@ def test_sector_stats_accuracy(test_db_session, multiple_officials_with_patterns
"""Test sector statistics are calculated correctly.""" """Test sector statistics are calculated correctly."""
session = test_db_session session = test_db_session
detector = PatternDetector(session) detector = PatternDetector(session)
sector_stats = detector.get_sector_timing_analysis(lookback_days=3650) sector_stats = detector.get_sector_timing_analysis(lookback_days=3650)
# Energy should have clean pattern (no alerts) # Energy should have clean pattern (no alerts)
if "Energy" in sector_stats: if "Energy" in sector_stats:
energy = sector_stats["Energy"] energy = sector_stats["Energy"]
@@ -313,14 +308,12 @@ def test_party_stats_completeness(test_db_session, multiple_officials_with_patte
"""Test party statistics completeness.""" """Test party statistics completeness."""
session = test_db_session session = test_db_session
detector = PatternDetector(session) detector = PatternDetector(session)
party_stats = detector.get_party_comparison(lookback_days=3650) party_stats = detector.get_party_comparison(lookback_days=3650)
# Check Democrats (Pelosi) # Check Democrats (Pelosi)
if "Democrat" in party_stats: if "Democrat" in party_stats:
dem = party_stats["Democrat"] dem = party_stats["Democrat"]
assert dem["official_count"] >= 1 assert dem["official_count"] >= 1
assert dem["total_trades"] >= 5 # Pelosi has 5 trades assert dem["total_trades"] >= 5 # Pelosi has 5 trades
assert dem["total_suspicious"] > 0 # Pelosi has suspicious trades assert dem["total_suspicious"] > 0 # Pelosi has suspicious trades