Author SHA1 Message Date
ilia 332e38dbe8 Merge pull request 'Add AGENTS.md orientation for Cursor agents' (#9) from chore/bootstrap-agents-md into main
CI / skip-ci-check (push) Successful in 34s
CI / secret-scan (push) Successful in 30s
CI / python-ci (push) Successful in 1m42s
2026-08-05 17:51:42 -05:00
ilia 68a14d5d19 Add AGENTS.md orientation for Cursor agents
CI / skip-ci-check (pull_request) Successful in 34s
CI / secret-scan (pull_request) Successful in 29s
CI / python-ci (pull_request) Successful in 2m57s
2026-08-05 18:14:15 -04:00
ilia c8a10b4270 Merge pull request 'Humanize README and docs tone' (#8) from docs/humanize-prose into main
CI / skip-ci-check (push) Successful in 32s
CI / secret-scan (push) Successful in 30s
CI / python-ci (push) Successful in 2m24s
2026-08-05 14:18:01 -05:00
ilia 662cb1b1b8 Humanize READMEs and guides for clearer, professional tone
CI / skip-ci-check (pull_request) Successful in 32s
CI / secret-scan (pull_request) Successful in 30s
CI / python-ci (pull_request) Successful in 2m5s
Align with project-template docs/writing-docs.md: plain openings, no
emoji decoration, archive scratch plans where they cluttered live docs.
2026-08-05 14:52:53 -04:00
ilia d790dc753b Merge pull request 'Add MIT LICENSE' (#7) from chore/add-license into main
CI / skip-ci-check (push) Successful in 19s
CI / secret-scan (push) Successful in 17s
CI / python-ci (push) Successful in 1m10s
2026-07-26 15:43:38 -05:00
ilia 821d4d58d8 Add MIT LICENSE
CI / skip-ci-check (pull_request) Successful in 18s
CI / secret-scan (pull_request) Successful in 17s
CI / python-ci (pull_request) Successful in 1m10s
2026-07-26 15:40:19 -05:00
ilia 1665885927 Merge pull request 'Fix lint debt and make CI gates honest' (#6) from chore/lint-debt-and-honest-ci into main
CI / skip-ci-check (push) Successful in 22s
CI / python-ci (push) Successful in 1m40s
CI / secret-scan (push) Successful in 19s
2026-07-26 14:47:32 -05:00
ilia 5df21d82f4 Fix lint debt and make CI gates honest
CI / skip-ci-check (pull_request) Successful in 19s
CI / secret-scan (pull_request) Successful in 19s
CI / python-ci (pull_request) Successful in 2m38s
- ruff: fix all 328 errors (autofix + manual); move config to [tool.ruff.lint]
- mypy: fix all errors (annotations, nullable-column guards, stale kwargs
  in weekly report caller)
- black: reformat src/tests so black --check passes
- CI: remove `|| true` from ruff/black/mypy/pytest steps in both
  .gitea and .github workflows; install project deps and add mypy
  step in gitea python-ci so gates actually run
- docs: move 14 root status/guide markdown files into docs/, update links
2026-07-26 15:41:07 -04:00
ilia 6e5e69ed10 ci: add local pre-commit gitleaks hook
CI / skip-ci-check (push) Successful in 11s
CI / secret-scan (push) Successful in 11s
CI / python-ci (push) Successful in 42s
2026-07-13 15:00:18 -05:00
ilia 781a45f1a3 Mark deploy and daily email verified (#4)
CI / skip-ci-check (push) Successful in 9s
CI / secret-scan (push) Successful in 8s
CI / python-ci (push) Successful in 22s
2026-07-12 10:38:51 -05:00
61 changed files with 1247 additions and 1340 deletions
+9 -9
View File
@@ -46,10 +46,16 @@ jobs:
python3 -m pip install --upgrade pip --break-system-packages
if [ -f requirements.txt ]; then pip install -r requirements.txt --break-system-packages; fi
if [ -f requirements-dev.txt ]; then pip install -r requirements-dev.txt --break-system-packages; fi
pip install bandit pip-audit ruff --break-system-packages
# Install the project + dev tools so lint/type/test gates run for real
pip install -e ".[dev]" --break-system-packages
pip install bandit pip-audit --break-system-packages
# Lint, type-check, and tests are hard gates — no `|| true`.
- name: Ruff lint
run: ruff check . || true
run: ruff check src tests
- name: Mypy
run: mypy src
- name: Bandit (advisory)
run: bandit -r . -q || true
@@ -58,13 +64,7 @@ jobs:
run: pip-audit -r requirements.txt 2>/dev/null || pip-audit 2>/dev/null || true
- name: Pytest
run: |
if [ -d tests ] || ls test_*.py *_test.py 2>/dev/null; then
pip install pytest --break-system-packages
pytest -q || true
else
echo "No tests found — skip"
fi
run: pytest -q
secret-scan:
needs: skip-ci-check
+4 -3
View File
@@ -34,14 +34,15 @@ jobs:
.venv/bin/pip install --upgrade pip
.venv/bin/pip install -e ".[dev]"
# Linters are hard gates — no `|| true`.
- name: Run linters
run: |
echo "Running ruff..."
.venv/bin/ruff check src/ tests/ || true
.venv/bin/ruff check src/ tests/
echo "Running black check..."
.venv/bin/black --check src/ tests/ || true
.venv/bin/black --check src/ tests/
echo "Running mypy..."
.venv/bin/mypy src/ --install-types --non-interactive || true
.venv/bin/mypy src/ --install-types --non-interactive
- name: Run tests with coverage
env:
+11
View File
@@ -1,6 +1,15 @@
# Homelab bootstrap — gitleaks allowlist (tests, examples, placeholders)
#
# IMPORTANT: `useDefault = true` is required — without it gitleaks loads ONLY
# this file (title + allowlist) with ZERO detection rules, so it would never
# flag a real secret. Fixed 2026-07 (security-hardening track); if you're
# re-pushing this template to a repo that already had the old version, that
# repo's secret scanning was a no-op until this lands.
title = "homelab gitea bootstrap"
[extend]
useDefault = true
[allowlist]
description = "Test fixtures and example configs are not production secrets"
paths = [
@@ -11,9 +20,11 @@ paths = [
'''(?i).*\.example\.(yml|yaml|env|json|toml)$''',
'''(?i).*vault\.example\.(yml|yaml)$''',
'''(?i).*\.env\.example$''',
'''(?i)CUSTOMIZATION_CHECKLIST\.md$''',
]
regexes = [
'''(?i)(invalid|fake|dummy|placeholder|example|changeme|change_me|not-a-real)''',
'''(?i)sk-or-invalid''',
'''(?i)msk-or-invalid''',
'''(?i)your_ssh_private_key_here''',
]
+20
View File
@@ -0,0 +1,20 @@
# AGENTS.md — POTE
Short orientation for Cursor agents. Prefer this over rediscovering the repo.
## Defaults
- Read `README.md` for run/install; use `make help` when a Makefile exists.
- Lint and tests are **hard gates** in CI — never add `|| true` to them.
- **Do not** commit secrets; app creds live in Infisical (`secrets.levkin.ca`, `/apps/POTE`).
- Multi-step or ambiguous work → Plan mode first, then Agent mode.
## Docs voice
READMEs/guides: `~/Documents/code/project-template/docs/writing-docs.md` (no emoji decoration).
## Close-out
1. Run the narrowest verify that fits (`make test`, `npm test`, `pytest`, CI workflow locally).
2. Public GitHub mirror (`Gitilia/*`): after README/docs release merges, sync mirrors — see `ansible/docs/guides/github-mirrors.md`.
3. Non-trivial PR: offer review; merge only when the user asks.
+21
View File
@@ -0,0 +1,21 @@
MIT License
Copyright (c) 2026 Ilia Dobkin
Permission is hereby granted, free of charge, to any person obtaining a copy
of this software and associated documentation files (the "Software"), to deal
in the Software without restriction, including without limitation the rights
to use, copy, modify, merge, publish, distribute, sublicense, and/or sell
copies of the Software, and to permit persons to whom the Software is
furnished to do so, subject to the following conditions:
The above copyright notice and this permission notice shall be included in all
copies or substantial portions of the Software.
THE SOFTWARE IS PROVIDED "AS IS", WITHOUT WARRANTY OF ANY KIND, EXPRESS OR
IMPLIED, INCLUDING BUT NOT LIMITED TO THE WARRANTIES OF MERCHANTABILITY,
FITNESS FOR A PARTICULAR PURPOSE AND NONINFRINGEMENT. IN NO EVENT SHALL THE
AUTHORS OR COPYRIGHT HOLDERS BE LIABLE FOR ANY CLAIM, DAMAGES OR OTHER
LIABILITY, WHETHER IN AN ACTION OF CONTRACT, TORT OR OTHERWISE, ARISING FROM,
OUT OF OR IN CONNECTION WITH THE SOFTWARE OR THE USE OR OTHER DEALINGS IN THE
SOFTWARE.
+37 -174
View File
@@ -1,204 +1,67 @@
# POTE Public Officials Trading Explorer
# POTE (Public Officials Trading Explorer)
**Research-only tool for tracking and analyzing public stock trades by government officials.**
Research tool for tracking publicly disclosed stock trades by government
officials (starting with U.S. Congress). Computes descriptive metrics and
risk/ethics flags from lawfully available public data.
⚠️ **Important**: This project is for personal research and transparency analysis only. It is **NOT** for investment advice or live trading.
Not investment advice. Not for live trading. Public disclosures only; data
may be delayed or incomplete. No claims about inside information.
## What is this?
POTE tracks stock trading activity of government officials (starting with U.S. Congress) using lawfully available public data sources. It computes research metrics, descriptive signals, and risk/ethics flags to help understand trading patterns.
## Key constraints
- **Public data only**: House Stock Watcher (free!), yfinance (free!), QuiverQuant/FMP (optional)
- **Research framing**: All outputs are descriptive analytics, not trading recommendations
- **No inside information claims**: We use public disclosures that may be delayed or incomplete
## Current Status
**PR1 Complete**: Project scaffold, DB models, price loader
**PR2 Complete**: Congressional trade ingestion (House Stock Watcher)
**PR3 Complete**: Security enrichment + deployment infrastructure
**PR4 Complete**: Phase 2 analytics - returns, benchmarks, performance metrics
**45+ passing tests, 88%+ coverage**
Status: active. Homelab LXC deploy documented under `docs/`.
## Quick start
**🚀 Already deployed?** See **[QUICKSTART.md](QUICKSTART.md)** for full usage guide!
**📦 Deploying?** See **[PROXMOX_QUICKSTART.md](PROXMOX_QUICKSTART.md)** for Proxmox LXC deployment (recommended).
**📧 Want automated reports?** See **[AUTOMATION_QUICKSTART.md](AUTOMATION_QUICKSTART.md)** for email reporting setup!
**🏠 Homelab deploy (LXC 236)?** See **[docs/HANDOFF-2026-05-27.md](docs/HANDOFF-2026-05-27.md)** for ops handoff and next steps.
### Local Development
```bash
# Install
git clone <your-repo>
cd pote
git clone <repository-url>
cd pote # or POTE
make install
source venv/bin/activate
# Run migrations
make migrate
python scripts/ingest_from_fixtures.py # offline sample
make test
```
# Ingest sample data (offline, for testing)
python scripts/ingest_from_fixtures.py
With network:
# Enrich securities with company info
python scripts/enrich_securities.py
# With internet:
```bash
python scripts/fetch_congressional_trades.py
python scripts/fetch_sample_prices.py
# Run tests
make test
# Lint & format
make lint format
```
### Production Deployment
Deploy: `bash scripts/proxmox_setup.sh` or `docker-compose up -d`.
Ops handoff: [docs/HANDOFF-2026-05-27.md](docs/HANDOFF-2026-05-27.md).
## Useful commands
```bash
# Proxmox LXC (Recommended - 5 minutes)
bash scripts/proxmox_setup.sh
# Docker
docker-compose up -d
```
## Tech stack
- **Language**: Python 3.10+
- **Database**: PostgreSQL or SQLite (dev)
- **Data**: House Stock Watcher (free!), yfinance (free!), QuiverQuant/FMP (optional)
- **Libraries**: SQLAlchemy, Alembic, pandas, numpy, httpx, yfinance, scikit-learn
- **Testing**: pytest (28 tests, 87%+ coverage)
## Documentation
**Getting Started**:
- [`README.md`](README.md) This file
- [`QUICKSTART.md`](QUICKSTART.md) **How to use your deployed POTE instance**
- [`STATUS.md`](STATUS.md) Current project status
- [`FREE_TESTING_QUICKSTART.md`](FREE_TESTING_QUICKSTART.md) Test for $0
- [`OFFLINE_DEMO.md`](OFFLINE_DEMO.md) Works without internet!
**Deployment**:
- [`PROXMOX_QUICKSTART.md`](PROXMOX_QUICKSTART.md) **Proxmox quick deployment (5 min)**
- [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md) **Automated reporting setup (5 min)**
- [`docs/07_deployment.md`](docs/07_deployment.md) Full deployment guide (all platforms)
- [`docs/08_proxmox_deployment.md`](docs/08_proxmox_deployment.md) Proxmox detailed guide
- [`docs/12_automation_and_reporting.md`](docs/12_automation_and_reporting.md) Automation & CI/CD guide
- [`Dockerfile`](Dockerfile) + [`docker-compose.yml`](docker-compose.yml) Docker setup
**Technical**:
- [`docs/00_mvp.md`](docs/00_mvp.md) MVP roadmap
- [`docs/01_architecture.md`](docs/01_architecture.md) Architecture
- [`docs/02_data_model.md`](docs/02_data_model.md) Database schema
- [`docs/03_data_sources.md`](docs/03_data_sources.md) Data sources
- [`docs/04_safety_ethics.md`](docs/04_safety_ethics.md) Research-only guardrails
- [`docs/05_dev_setup.md`](docs/05_dev_setup.md) Dev conventions
- [`docs/06_free_testing_data.md`](docs/06_free_testing_data.md) Testing strategies
**PR Summaries**:
- [`docs/PR1_SUMMARY.md`](docs/PR1_SUMMARY.md) Scaffold + price loader
- [`docs/PR2_SUMMARY.md`](docs/PR2_SUMMARY.md) Congressional trades
- [`docs/PR3_SUMMARY.md`](docs/PR3_SUMMARY.md) Enrichment + deployment
- [`docs/PR4_SUMMARY.md`](docs/PR4_SUMMARY.md) **Analytics foundation (returns, benchmarks, metrics)**
## What's Working Now
- ✅ SQLAlchemy models for officials, securities, trades, prices
- ✅ Alembic migrations
- ✅ Price loader with yfinance (idempotent, upsert)
- ✅ Congressional trade ingestion from House Stock Watcher (FREE!)
- ✅ Security enrichment (company names, sectors, industries)
- ✅ ETL to populate officials & trades tables
- ✅ Docker + deployment infrastructure
- ✅ 93 passing tests with 88%+ coverage
- ✅ Linting (ruff + mypy) all green
- ✅ Works 100% offline with fixtures
- ✅ Real-time market monitoring & alert system
- ✅ Disclosure timing correlation engine
- ✅ Pattern detection & comparative analysis
- ✅ Automated email reporting (daily/weekly)
- ✅ CI/CD pipeline (GitHub/Gitea Actions)
## What You Can Do Now
### Analyze Performance
```bash
# Analyze specific official
python scripts/analyze_official.py "Nancy Pelosi" --window 90
# System-wide analysis
python scripts/calculate_all_returns.py
```
### Market Monitoring
```bash
# Run market scan
python scripts/monitor_market.py --scan
# Analyze timing of recent disclosures
python scripts/analyze_disclosure_timing.py --recent 7
# Generate pattern report
python scripts/generate_pattern_report.py --days 365
```
### Automated Reporting
```bash
# Set up daily/weekly email reports (5 minutes!)
./scripts/setup_cron.sh
# Send manual report
python scripts/send_daily_report.py --to your@email.com
```
### Add More Data
```bash
# Manual entry
python scripts/add_custom_trades.py
## Stack
# CSV import
python scripts/scrape_alternative_sources.py import trades.csv
```
Python 3.10+, SQLAlchemy, Alembic, pandas/numpy, httpx, yfinance.
PostgreSQL or SQLite for local. pytest + ruff + mypy.
## System Architecture
Data sources: House Stock Watcher, yfinance; QuiverQuant/FMP optional.
POTE now includes a complete 3-phase monitoring system:
## Docs
**Phase 1: Real-Time Market Monitoring**
- Tracks ~50 most-traded congressional stocks
- Detects unusual volume, price spikes, volatility
- Logs all alerts with timestamps and severity
| Doc | Purpose |
|-----|---------|
| [docs/QUICKSTART.md](docs/QUICKSTART.md) | Using a deployed instance |
| [docs/PROXMOX_QUICKSTART.md](docs/PROXMOX_QUICKSTART.md) | Proxmox LXC |
| [docs/AUTOMATION_QUICKSTART.md](docs/AUTOMATION_QUICKSTART.md) | Email reports |
| [docs/01_architecture.md](docs/01_architecture.md) | Architecture |
| [docs/02_data_model.md](docs/02_data_model.md) | Schema |
| [docs/04_safety_ethics.md](docs/04_safety_ethics.md) | Research guardrails |
| [docs/07_deployment.md](docs/07_deployment.md) | Full deploy |
| [docs/00_mvp.md](docs/00_mvp.md) | Roadmap |
**Phase 2: Disclosure Correlation**
- Matches trades with prior market alerts (30-45 day lookback)
- Calculates "timing advantage score" (0-100)
- Identifies suspicious timing patterns
PR writeups live under `docs/archive/` when moved.
**Phase 3: Pattern Detection**
- Ranks officials by consistent suspicious timing
- Analyzes by ticker, sector, and political party
- Generates comprehensive reports
## License
**Full Documentation**: See [`MONITORING_SYSTEM_COMPLETE.md`](MONITORING_SYSTEM_COMPLETE.md)
## Next Steps
- [ ] Signals: "follow_research", "avoid_risk", "watch" with confidence scores
- [ ] Clustering: group officials by trading behavior patterns
- [ ] API: FastAPI backend for queries
- [ ] Dashboard: React/Streamlit visualization
See [`docs/00_mvp.md`](docs/00_mvp.md) for the full roadmap.
---
**License**: MIT (for research/educational use only)
**Disclaimer**: Not investment advice. Use public data only. No claims about inside information.
MIT for research/educational use. Not investment advice.
+4 -4
View File
@@ -6,7 +6,7 @@ Great question! Here are multiple strategies for testing the full pipeline **wit
---
## Strategy 1: Mock/Fixture Data (Current Approach )
## Strategy 1: Mock/Fixture Data (Current Approach )
**What we already have:**
- `tests/conftest.py` creates in-memory SQLite DB with sample officials, securities, trades
@@ -99,7 +99,7 @@ def test_etl_with_real_data():
---
## Strategy 4: Hybrid Testing (Recommended 🌟)
## Strategy 4: Hybrid Testing (Recommended )
**Combine all strategies**:
@@ -164,7 +164,7 @@ def test_etl_with_real_data():
| Source | Free Tier | Paid Tier | Best For |
|--------|-----------|-----------|----------|
| **yfinance** | Unlimited | N/A | Prices (already working ) |
| **yfinance** | Unlimited | N/A | Prices (already working ) |
| **House Stock Watcher** | Unlimited scraping | N/A | Free trades (best option) |
| **Quiver Free** | 500 calls/mo | $30/mo (5k calls) | Testing, not production |
| **FMP Free** | 250 calls/day | $15/mo | Alternative for trades |
@@ -222,5 +222,5 @@ class HouseWatcherClient:
}
```
Let me know if you want me to implement this scraper now for PR2! 🚀
Let me know if you want me to implement this scraper now for PR2!
+2 -2
View File
@@ -4,14 +4,14 @@
POTE can be deployed in several ways depending on your needs:
1. **Local Development** (SQLite) - What you have now
1. **Local Development** (SQLite) - What you have now
2. **Single Server** (PostgreSQL + cron jobs)
3. **Docker** (Containerized, easy to move)
4. **Cloud** (AWS/GCP/Azure with managed DB)
---
## Option 1: Local Development (Current Setup)
## Option 1: Local Development (Current Setup)
**You're already running this!**
+16 -16
View File
@@ -2,17 +2,17 @@
## Why Proxmox is Perfect for POTE
**Full control** - Your hardware, your rules
**No monthly costs** - Just electricity
**Isolated VMs/LXC** - Clean environments
**Snapshots** - Easy rollback if needed
**Resource efficient** - Run alongside other services
**Full control** - Your hardware, your rules
**No monthly costs** - Just electricity
**Isolated VMs/LXC** - Clean environments
**Snapshots** - Easy rollback if needed
**Resource efficient** - Run alongside other services
---
## Deployment Options on Proxmox
### Option 1: LXC Container (Recommended)
### Option 1: LXC Container (Recommended)
**Pros**: Lightweight, fast, efficient resource usage
**Cons**: Linux only (fine for POTE)
@@ -501,14 +501,14 @@ vs.
## Next Steps
1. Create LXC container
2. Install dependencies
3. Setup PostgreSQL
4. Deploy POTE
5. Configure cron jobs
6. Setup backups
7. Build Phase 2 (Analytics)
8. Add FastAPI dashboard (optional)
1. Create LXC container
2. Install dependencies
3. Setup PostgreSQL
4. Deploy POTE
5. Configure cron jobs
6. Setup backups
7. ⏭ Build Phase 2 (Analytics)
8. ⏭ Add FastAPI dashboard (optional)
---
@@ -583,7 +583,7 @@ EOF
sudo -u poteapp mkdir -p /home/poteapp/logs
echo ""
echo " Setup complete!"
echo " Setup complete!"
echo ""
echo "Next steps:"
echo "1. su - poteapp"
@@ -600,5 +600,5 @@ chmod +x proxmox_setup.sh
---
**Your Proxmox setup gives you enterprise-grade infrastructure at hobby costs!** 🚀
**Your Proxmox setup gives you enterprise-grade infrastructure at hobby costs!**
+5 -5
View File
@@ -137,13 +137,13 @@ python scripts/fetch_sample_prices.py
## Data Sources
### Currently Working:
- yfinance (prices, company info)
- Manual entry
- CSV import
- Fixture files (testing)
- yfinance (prices, company info)
- Manual entry
- CSV import
- Fixture files (testing)
### Currently Down:
- House Stock Watcher API (domain issues)
- House Stock Watcher API (domain issues)
### Future Options:
- QuiverQuant (requires $30/month subscription)
+19 -19
View File
@@ -8,10 +8,10 @@
### **Reality Check: No Real-Time Data Exists**
**Federal Law (STOCK Act):**
- 📅 Congress members have **30-45 days** to disclose trades
- 📅 Disclosures are filed as **Periodic Transaction Reports (PTRs)**
- 📅 Public databases update **after** filing (usually next day)
- 📅 **No real-time feed exists by design**
- Congress members have **30-45 days** to disclose trades
- Disclosures are filed as **Periodic Transaction Reports (PTRs)**
- Public databases update **after** filing (usually next day)
- **No real-time feed exists by design**
**Example Timeline:**
```
@@ -25,15 +25,15 @@ Feb 17, 2024 → Your system fetches it
Since trades appear in batches (not continuously), **running once per day is optimal**:
**Daily (7 AM)** - Catches overnight filings
**After market close** - Prices are final
**Low server load** - Off-peak hours
**Hourly** - Wasteful, no new data
**Real-time** - Impossible, not how disclosures work
**Daily (7 AM)** - Catches overnight filings
**After market close** - Prices are final
**Low server load** - Off-peak hours
**Hourly** - Wasteful, no new data
**Real-time** - Impossible, not how disclosures work
---
## 🤖 Automated Setup Options
## Automated Setup Options
### **Option 1: Cron Job (Linux/Proxmox) - Recommended**
@@ -149,7 +149,7 @@ Or from anywhere:
---
## 📊 What Gets Updated?
## What Gets Updated?
### **1. Congressional Trades**
**Script:** `fetch_congressional_trades.py`
@@ -182,7 +182,7 @@ Or from anywhere:
---
## ⚙️ Customizing the Schedule
## Customizing the Schedule
### **Different Frequencies**
@@ -215,7 +215,7 @@ Or from anywhere:
---
## 📧 Email Notifications (Optional)
## Email Notifications (Optional)
### **Setup Email Alerts**
@@ -310,7 +310,7 @@ python scripts/email_summary.py
---
## 🔍 Monitoring & Logging
## Monitoring & Logging
### **Check Cron Job Status**
@@ -355,7 +355,7 @@ Add to `/etc/logrotate.d/pote`:
---
## 🚨 Handling Failures
## Handling Failures
### **What If House Stock Watcher Is Down?**
@@ -363,7 +363,7 @@ The script is designed to continue even if one step fails:
```bash
# Script continues and logs warnings
⚠️ WARNING: Failed to fetch congressional trades
WARNING: Failed to fetch congressional trades
This is likely because House Stock Watcher API is down
Continuing with other steps...
```
@@ -399,7 +399,7 @@ for attempt in range(MAX_RETRIES):
---
## 📈 Performance Optimization
## Performance Optimization
### **Batch Processing**
@@ -439,7 +439,7 @@ CREATE INDEX IF NOT EXISTS ix_prices_security_id ON prices(security_id);
---
## 🎯 Recommended Setup
## Recommended Setup
### **For Proxmox Production:**
@@ -475,7 +475,7 @@ pote-update
---
## 📝 Summary
## Summary
### **Key Points:**
+67 -67
View File
@@ -1,13 +1,13 @@
# Live Market Monitoring + Congressional Trading Analysis
## 🎯 What's Possible vs Impossible
## What's Possible vs Impossible
### **NOT Possible:**
### **NOT Possible:**
- Identify WHO is buying/selling in real-time
- Match live trades to specific Congress members
- See congressional trades before they're disclosed
### **IS Possible:**
### **IS Possible:**
- Track unusual market activity in real-time
- Monitor stocks Congress members historically trade
- Compare unusual activity to later disclosures
@@ -15,7 +15,7 @@
---
## 🔄 **Two-Phase Monitoring System**
## **Two-Phase Monitoring System**
### **Phase 1: Real-Time Market Monitoring**
Monitor unusual activity in stocks Congress trades:
@@ -33,7 +33,7 @@ When disclosures come in:
---
## 📊 **Implementation: Watchlist-Based Monitoring**
## **Implementation: Watchlist-Based Monitoring**
### **Concept:**
@@ -57,38 +57,38 @@ Step 3: When Disclosure Appears (30-45 days later)
---
## 🛠️ **Data Sources for Live Market Monitoring**
## **Data Sources for Live Market Monitoring**
### **Free/Low-Cost Options:**
1. **Yahoo Finance (yfinance)**
- Real-time quotes (15-min delay free)
- Historical options data
- Volume data
- Not true real-time for options flow
- Real-time quotes (15-min delay free)
- Historical options data
- Volume data
- Not true real-time for options flow
2. **Unusual Whales API**
- Options flow data
- Unusual activity alerts
- 💰 Paid ($50-200/month)
- Options flow data
- Unusual activity alerts
- Paid ($50-200/month)
- https://unusualwhales.com/
3. **Tradier API**
- Real-time market data
- Options chains
- 💰 Paid but affordable ($10-50/month)
- Real-time market data
- Options chains
- Paid but affordable ($10-50/month)
- https://tradier.com/
4. **FlowAlgo**
- Options flow tracking
- Dark pool data
- 💰 Paid ($99-399/month)
- Options flow tracking
- Dark pool data
- Paid ($99-399/month)
- https://www.flowalgo.com/
5. **Polygon.io**
- Real-time stock data
- Options data
- 💰 Free tier + paid plans
- Real-time stock data
- Options data
- Free tier + paid plans
- https://polygon.io/
### **Best Free Option: Build Your Own with yfinance**
@@ -97,7 +97,7 @@ Track volume/price changes every 5 minutes for congressional watchlist tickers.
---
## 💡 **Practical Hybrid System**
## **Practical Hybrid System**
### **What We Can Build:**
@@ -141,37 +141,37 @@ for disclosure in new_disclosures:
---
## 📈 **Example: Nancy Pelosi NVDA Trade Analysis**
## **Example: Nancy Pelosi NVDA Trade Analysis**
### **Timeline:**
```
Nov 10, 2024:
🔔 ALERT: NVDA unusual call options activity
ALERT: NVDA unusual call options activity
Volume: 10x average
Strike: $500 (2 weeks out)
Nov 15, 2024:
💰 Someone buys NVDA (unknown who at the time)
Someone buys NVDA (unknown who at the time)
Nov 18, 2024:
📰 NVDA announces new AI chip
📈 Stock jumps 15%
NVDA announces new AI chip
Stock jumps 15%
Dec 15, 2024:
📋 Disclosure: Nancy Pelosi bought NVDA on Nov 15
Disclosure: Nancy Pelosi bought NVDA on Nov 15
Value: $15,001-$50,000
ANALYSIS:
She bought AFTER unusual options activity (Nov 10)
She bought BEFORE announcement (Nov 18)
Timing: 3 days before major news
🚩 Flag: Investigate if announcement was public knowledge
She bought AFTER unusual options activity (Nov 10)
She bought BEFORE announcement (Nov 18)
⏱ Timing: 3 days before major news
Flag: Investigate if announcement was public knowledge
```
---
## 🎯 **Recommended Approach**
## **Recommended Approach**
### **Phase 1: Build Congressional Ticker Watchlist**
@@ -230,12 +230,12 @@ def monitor_tickers(tickers, interval_minutes=5):
# Check for unusual volume
avg_volume = current['Volume'].mean()
if latest['Volume'] > avg_volume * 3:
alert(f"🔔 {ticker}: Unusual volume spike!")
alert(f" {ticker}: Unusual volume spike!")
# Check for price movement
price_change = (latest['Close'] - current['Open'].iloc[0]) / current['Open'].iloc[0]
if abs(price_change) > 0.05: # 5% move
alert(f"📈 {ticker}: {price_change:.2%} move today!")
alert(f" {ticker}: {price_change:.2%} move today!")
except Exception as e:
print(f"Error monitoring {ticker}: {e}")
@@ -290,31 +290,31 @@ def analyze_disclosure_timing(disclosure):
---
## 🚨 **Realistic Expectations**
## **Realistic Expectations**
### **What This System Will Do:**
Monitor stocks Congress members historically trade
Alert on unusual market activity in those stocks
Retroactively correlate disclosures with earlier alerts
Identify timing patterns and potential advantages
Build database of congressional trading patterns
Monitor stocks Congress members historically trade
Alert on unusual market activity in those stocks
Retroactively correlate disclosures with earlier alerts
Identify timing patterns and potential advantages
Build database of congressional trading patterns
### **What This System WON'T Do:**
Identify WHO is buying in real-time
Give you advance notice of congressional trades
Provide real-time inside information
Allow you to "front-run" Congress
Identify WHO is buying in real-time
Give you advance notice of congressional trades
Provide real-time inside information
Allow you to "front-run" Congress
### **Legal & Ethical:**
All data is public
Analysis is retrospective
For research and transparency
Not market manipulation
Cannot and should not be used to replicate potentially illegal trades
All data is public
Analysis is retrospective
For research and transparency
Not market manipulation
Cannot and should not be used to replicate potentially illegal trades
---
## 📊 **Proposed Implementation**
## **Proposed Implementation**
### **New Scripts to Create:**
@@ -365,15 +365,15 @@ CREATE TABLE disclosure_timing_analysis (
---
## 🎯 **Summary**
## **Summary**
### **Your Question:**
> "Can we read live trades being made and compare them to a name?"
### **Answer:**
**No** - Live trades are anonymous, can't identify individuals
**No** - Live trades are anonymous, can't identify individuals
**BUT** - You CAN:
**BUT** - You CAN:
1. Monitor unusual activity in stocks Congress trades
2. Log these alerts in real-time
3. When disclosures appear (30-45 days later), correlate them
@@ -381,25 +381,25 @@ CREATE TABLE disclosure_timing_analysis (
5. Build patterns database of timing and performance
### **This Gives You:**
- Transparency on timing advantages
- Pattern detection across officials
- Research-grade analysis
- Historical correlation data
- Transparency on timing advantages
- Pattern detection across officials
- Research-grade analysis
- Historical correlation data
### **This Does NOT Give You:**
- Real-time identity of traders
- Advance notice of congressional trades
- Ability to "front-run" disclosures
- Real-time identity of traders
- Advance notice of congressional trades
- Ability to "front-run" disclosures
---
## 🚀 **Would You Like Me To Build This?**
## **Would You Like Me To Build This?**
I can create:
1. Real-time monitoring system for congressional tickers
2. Alert logging and analysis
3. Timing correlation when disclosures appear
4. Pattern detection and reporting
1. Real-time monitoring system for congressional tickers
2. Alert logging and analysis
3. Timing correlation when disclosures appear
4. Pattern detection and reporting
This would be **Phase 2.5** of POTE - the "timing analysis" module.
+5 -5
View File
@@ -205,18 +205,18 @@ Output:
POTE HEALTH CHECK
============================================================
Timestamp: 2025-12-15T10:30:00
Overall Status: OK
Overall Status: OK
Database Connection: Database connection successful
Data Freshness: Data is fresh (2 days old)
Database Connection: Database connection successful
Data Freshness: Data is fresh (2 days old)
latest_trade_date: 2025-12-13
Data Counts: Database has 1,234 trades
Data Counts: Database has 1,234 trades
officials: 45
securities: 123
trades: 1,234
prices: 12,345
market_alerts: 567
Recent Alerts: 23 alerts in last 24 hours
Recent Alerts: 23 alerts in last 24 hours
============================================================
```
+35 -35
View File
@@ -22,18 +22,18 @@ chmod 600 .env
chown poteapp:poteapp .env
```
### Pros
### Pros
- Simple, works immediately
- No additional setup
- Standard practice for Python projects
### ⚠️ Cons
### Cons
- Secrets stored in plain text on disk
- Risk if server is compromised
- No audit trail
### 🔒 Security Checklist
- [ ] `.env` in `.gitignore` (already done )
### Security Checklist
- [ ] `.env` in `.gitignore` (already done )
- [ ] File permissions: `chmod 600 .env`
- [ ] Never commit to git
- [ ] Backup securely (encrypted)
@@ -76,12 +76,12 @@ sudo chmod 600 /etc/systemd/system/pote.service
sudo systemctl daemon-reload
```
### Pros
### Pros
- Secrets not in git or project directory
- Standard Linux practice
- Works with systemd timers
### ⚠️ Cons
### Cons
- Still visible in `systemctl show`
- Requires root to edit
@@ -128,12 +128,12 @@ source .env
python scripts/send_daily_report.py
```
### Pros
### Pros
- Secrets separate from code
- Easy to rotate
- Can be backed up separately
### ⚠️ Cons
### Cons
- Extra file to manage
- Still plain text
@@ -191,12 +191,12 @@ class Settings(BaseSettings):
smtp_password: str = Field(default_factory=lambda: get_secret("SMTP_PASSWORD"))
```
### Pros
### Pros
- Docker-native solution
- Encrypted in Swarm mode
- Never in logs
### ⚠️ Cons
### Cons
- Requires Docker
- More complex setup
@@ -228,13 +228,13 @@ secrets = client.secrets.kv.v2.read_secret_version(path='pote')
smtp_password = secrets['data']['data']['smtp_password']
```
### Pros
### Pros
- Centralized secrets management
- Audit logs
- Dynamic secrets
- Access control
### ⚠️ Cons
### Cons
- Complex setup
- Requires Vault infrastructure
- Overkill for single user
@@ -260,14 +260,14 @@ env:
DATABASE_URL: postgresql://user:${{ secrets.DB_PASSWORD }}@postgres/db
```
### ⚠️ Important
### Important
- **Only for CI/CD pipelines**
- **NOT for deployed servers**
- Secrets are injected during workflow runs
---
## 🎯 Recommendation for Your Setup
## Recommendation for Your Setup
### Personal/Research Use (Current)
@@ -305,9 +305,9 @@ gpg -c .env # Creates .env.gpg
---
## 🔒 General Security Best Practices
## General Security Best Practices
### DO
### DO
- Use strong, unique passwords
- Restrict file permissions (`chmod 600`)
@@ -316,7 +316,7 @@ gpg -c .env # Creates .env.gpg
- Use encrypted backups
- Audit who has server access
### DON'T
### DON'T
- Commit secrets to git (even private repos)
- Store passwords in code
@@ -327,7 +327,7 @@ gpg -c .env # Creates .env.gpg
---
## 🧪 Test Your Security
## Test Your Security
### Check if `.env` is protected
@@ -352,7 +352,7 @@ git log --all --full-history --source --pickaxe-all -S 'smtp_password'
---
## 🔄 Password Rotation Procedure
## Password Rotation Procedure
### Every 90 days (or if compromised):
@@ -369,32 +369,32 @@ git log --all --full-history --source --pickaxe-all -S 'smtp_password'
---
## 📊 Security Level Comparison
## Security Level Comparison
| Level | Method | Effort | Protection |
|-------|--------|--------|------------|
| 🔓 Basic | `.env` (default perms) | None | Low |
| 🔒 Good | `.env` (chmod 600) | 1 min | Medium |
| 🔒 Better | Environment variables | 10 min | Good |
| 🔒 Better | Separate secrets file | 10 min | Good |
| 🔐 Best | Docker Secrets | 30 min | Very Good |
| 🔐 Best | Vault | 2+ hours | Excellent |
| Basic | `.env` (default perms) | None | Low |
| Good | `.env` (chmod 600) | 1 min | Medium |
| Better | Environment variables | 10 min | Good |
| Better | Separate secrets file | 10 min | Good |
| Best | Docker Secrets | 30 min | Very Good |
| Best | Vault | 2+ hours | Excellent |
---
## 🎯 Your Current Status
## Your Current Status
**Already secure enough for personal use:**
- `.env` in `.gitignore`
- Not committed to git
- Local server only
**Already secure enough for personal use:**
- `.env` in `.gitignore`
- Not committed to git
- Local server only
⚠️ **Recommended improvement (2 minutes):**
**Recommended improvement (2 minutes):**
```bash
chmod 600 .env
```
🔐 **Optional (if paranoid):**
**Optional (if paranoid):**
- Use separate secrets file in `/etc/pote/`
- Encrypt backups with GPG
- Set up password rotation schedule
@@ -405,9 +405,9 @@ chmod 600 .env
**For your levkin.ca setup:**
1. **Current approach (`.env` file) is fine**
1. **Current approach (`.env` file) is fine**
2. **Add `chmod 600 .env`** for better security (2 minutes)
3. **Don't commit `.env` to git** (already protected )
3. **Don't commit `.env` to git** (already protected )
4. **Consider upgrading to environment variables** if you deploy to production
Your current setup is **appropriate for a personal research project**. Don't over-engineer it unless you have specific compliance requirements or a team.
@@ -62,7 +62,7 @@ Follow the prompts:
2. Choose daily report time (recommend 6 AM)
3. Confirm
That's it! 🎉
That's it!
---
@@ -112,7 +112,7 @@ Run the daily script manually to test:
./scripts/automated_daily_run.sh
```
Check if email arrived! 📧
Check if email arrived!
---
@@ -244,5 +244,5 @@ Add to cron for regular health checks:
---
**You're all set! POTE will now run automatically and send you daily/weekly reports. 🚀**
**You're all set! POTE will now run automatically and send you daily/weekly reports. **
@@ -1,17 +1,17 @@
# POTE Deployment & Automation Guide
## 🎯 Quick Answer to Your Questions
## Quick Answer to Your Questions
### After Deployment, What Happens?
**By default: NOTHING automatic happens.** You need to set up automation.
The deployed system is:
- Running (database, code installed)
- Accessible via SSH at your Proxmox IP
- NOT fetching data automatically
- NOT sending reports automatically
- NOT monitoring markets automatically
- Running (database, code installed)
- Accessible via SSH at your Proxmox IP
- NOT fetching data automatically
- NOT sending reports automatically
- NOT monitoring markets automatically
**You must either:**
1. **Run scripts manually** when you want updates, OR
@@ -19,7 +19,7 @@ The deployed system is:
---
## 🚀 Option 1: Automated Email Reports (Recommended)
## Option 1: Automated Email Reports (Recommended)
### What You Get
@@ -53,13 +53,13 @@ Run the interactive setup:
Follow prompts:
1. Enter your email address
2. Choose report time (default: 6 AM)
3. Done!
3. Done!
**See full guide:** [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md)
---
## 📍 Option 2: Access Reports via IP (No Email)
## Option 2: Access Reports via IP (No Email)
If you don't want email, you can:
@@ -107,7 +107,7 @@ python scripts/health_check.py
---
## 🌐 Option 3: Build a Web Interface (Future)
## Option 3: Build a Web Interface (Future)
Currently, POTE is **command-line only**. No web UI yet.
@@ -122,16 +122,16 @@ For now, use SSH or email reports.
---
## 🔄 Do You Need CI/CD Pipelines?
## Do You Need CI/CD Pipelines?
### What the Pipeline Does
The included CI/CD pipeline (`.github/workflows/ci.yml`) runs on **every git push**:
1. Lint & test (93 tests)
2. Security scanning
3. Dependency scanning
4. Docker build test
1. Lint & test (93 tests)
2. Security scanning
3. Dependency scanning
4. Docker build test
### Should You Use It?
@@ -174,18 +174,18 @@ docker build -t pote:test .
---
## 📊 Comparison of Options
## Comparison of Options
| Method | Pros | Cons | Best For |
|--------|------|------|----------|
| **Automated Email** | Convenient<br> No SSH needed<br> Daily/weekly updates | Requires SMTP setup | Most users |
| **SSH + Manual Scripts** | Full control<br> No email needed | Manual work<br> Must remember to run | Power users |
| **Saved Reports (SSH access)** | Automated<br> No email | Must SSH to view | Users without email |
| **Web Interface** | User-friendly | Not implemented yet | Future |
| **Automated Email** | Convenient<br> No SSH needed<br> Daily/weekly updates | Requires SMTP setup | Most users |
| **SSH + Manual Scripts** | Full control<br> No email needed | Manual work<br> Must remember to run | Power users |
| **Saved Reports (SSH access)** | Automated<br> No email | Must SSH to view | Users without email |
| **Web Interface** | User-friendly | Not implemented yet | Future |
---
## 🛠️ Your Ansible Pipeline
## Your Ansible Pipeline
Your existing Ansible CI/CD pipeline is **NOT directly usable** for POTE because:
@@ -197,11 +197,11 @@ Your existing Ansible CI/CD pipeline is **NOT directly usable** for POTE because
**Concepts from your pipeline that ARE used in POTE's CI/CD:**
- Security scanning (Trivy, Bandit instead of Gitleaks)
- Dependency scanning (Trivy instead of npm audit)
- SAST scanning (Bandit instead of Semgrep)
- Container scanning (Docker build test)
- Workflow summary generation
- Security scanning (Trivy, Bandit instead of Gitleaks)
- Dependency scanning (Trivy instead of npm audit)
- SAST scanning (Bandit instead of Semgrep)
- Container scanning (Docker build test)
- Workflow summary generation
**The POTE pipeline (`.github/workflows/ci.yml`) already includes all of these!**
@@ -217,7 +217,7 @@ Your Gitea server can run the POTE pipeline using Gitea Actions:
---
## 📧 Email Setup Examples
## Email Setup Examples
### Gmail (Most Common)
@@ -258,7 +258,7 @@ REPORT_RECIPIENTS=admin@yourdomain.com
---
## Recommended Setup for Most Users
## Recommended Setup for Most Users
1. **Deploy to Proxmox** (5 min)
```bash
@@ -283,7 +283,7 @@ REPORT_RECIPIENTS=admin@yourdomain.com
---
## 🔍 Monitoring & Health Checks
## Monitoring & Health Checks
### Add System Health Monitoring
@@ -310,17 +310,17 @@ python scripts/health_check.py
---
## 📚 Full Documentation
## Full Documentation
- **Automation Setup**: [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md)
- **Deployment**: [`PROXMOX_QUICKSTART.md`](PROXMOX_QUICKSTART.md)
- **Usage**: [`QUICKSTART.md`](QUICKSTART.md)
- **Automation Setup**: [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md)
- **Deployment**: [`PROXMOX_QUICKSTART.md`](PROXMOX_QUICKSTART.md)
- **Usage**: [`QUICKSTART.md`](QUICKSTART.md)
- **Detailed Automation Guide**: [`docs/12_automation_and_reporting.md`](docs/12_automation_and_reporting.md)
- **Monitoring System**: [`MONITORING_SYSTEM_COMPLETE.md`](MONITORING_SYSTEM_COMPLETE.md)
---
## 🎉 Summary
## Summary
**After deployment:**
- Reports are NOT sent automatically by default
@@ -334,5 +334,5 @@ python scripts/health_check.py
1. Deploy to Proxmox
2. Run `./scripts/setup_cron.sh`
3. Receive daily/weekly email reports
4. Done! 🚀
4. Done!
+10 -10
View File
@@ -2,7 +2,7 @@
Homelab POTE sends via Mailcow **`mail.levkine.ca`** using the shared **`alerts@levkine.ca`** mailbox (same as Kuma/Beszel). See ansible `docs/guides/smtp-inventory.md`.
## Configuration Done
## Configuration Done
The `.env` file has been created with these settings:
@@ -15,7 +15,7 @@ FROM_EMAIL=alerts@levkine.ca
REPORT_RECIPIENTS=idobkin@gmail.com
```
## 🔑 Next Steps
## Next Steps
### 1. Add Your Password
@@ -38,7 +38,7 @@ python scripts/send_daily_report.py --to test@levkin.ca --test-smtp
If successful, you'll see:
```
SMTP connection test successful!
Daily report sent successfully!
Daily report sent successfully!
```
And you should receive a test email at `test@levkin.ca`!
@@ -57,7 +57,7 @@ This will:
- Schedule daily reports (default: 6 AM)
- Schedule weekly reports (Sundays at 8 AM)
## 📧 Email Server Details (For Reference)
## Email Server Details (For Reference)
Based on your Thunderbird setup:
@@ -76,10 +76,10 @@ Based on your Thunderbird setup:
POTE only uses **SMTP (outgoing)** to send reports.
## 🔒 Security Notes
## Security Notes
1. **Never commit `.env` to git!**
- Already in `.gitignore`
- Already in `.gitignore`
- Contains sensitive password
2. **Password Security:**
@@ -92,7 +92,7 @@ POTE only uses **SMTP (outgoing)** to send reports.
chmod 600 .env # Only owner can read/write
```
## 🎯 Change Recipients
## Change Recipients
To send reports to different email addresses (not just test@levkin.ca):
@@ -109,14 +109,14 @@ python scripts/send_daily_report.py --to someone-else@example.com
# The FROM address will still be test@levkin.ca
```
## Testing Checklist
## Testing Checklist
- [ ] Updated `.env` with your actual password
- [ ] Run `python scripts/send_daily_report.py --to test@levkin.ca --test-smtp`
- [ ] Checked inbox at test@levkin.ca (check spam folder!)
- [ ] If successful, run `./scripts/setup_cron.sh` to automate
## 🐛 Troubleshooting
## Troubleshooting
### Error: "SMTP connection failed"
@@ -139,5 +139,5 @@ Check:
---
**You're all set! POTE will send reports from test@levkin.ca 📧**
**You're all set! POTE will send reports from test@levkin.ca **
@@ -2,14 +2,14 @@
## TL;DR: You can test everything for $0
### Already Working (PR1 )
### Already Working (PR1 )
- **Price data**: `yfinance` (free, unlimited)
- **Unit tests**: Mocked data in `tests/` (15 passing tests)
- **Coverage**: 87% without any paid APIs
### For PR2 (Congressional Trades) - FREE Options
#### Best Option: House Stock Watcher 🌟
#### Best Option: House Stock Watcher
```bash
# No API key needed, just scrape their public JSON
curl https://housestockwatcher.com/api/all_transactions
@@ -44,10 +44,10 @@ python scripts/fetch_house_watcher_sample.py # We'll build this in PR2
### What You DON'T Need to Pay For
QuiverQuant Pro ($30/mo) - free tier is enough for dev/testing
Financial Modeling Prep paid tier - free tier works
Any paid database hosting - SQLite works great locally
Any cloud services - runs 100% locally
QuiverQuant Pro ($30/mo) - free tier is enough for dev/testing
Financial Modeling Prep paid tier - free tier works
Any paid database hosting - SQLite works great locally
Any cloud services - runs 100% locally
### When You MIGHT Want Paid (Way Later)
@@ -56,7 +56,7 @@ python scripts/fetch_house_watcher_sample.py # We'll build this in PR2
- Multiple concurrent users on a dashboard
- Commercial use (check each API's terms)
**For personal research? Stay free forever. 🎉**
**For personal research? Stay free forever. **
---
@@ -1,20 +1,20 @@
# 🔐 Gitea Secrets Guide for POTE
# Gitea Secrets Guide for POTE
## YES! You Can Store Passwords in Gitea
## YES! You Can Store Passwords in Gitea
Gitea has a **Secrets** feature (like GitHub Actions secrets) that lets you store passwords securely and use them in:
1. **CI/CD pipelines** (Gitea Actions workflows)
2. **Deployment workflows**
1. **CI/CD pipelines** (Gitea Actions workflows)
2. **Deployment workflows**
**BUT NOT:**
- Directly in your running application on Proxmox
- Accessed by scripts outside of workflows
- Directly in your running application on Proxmox
- Accessed by scripts outside of workflows
---
## 🎯 What Gitea Secrets Are Good For
## What Gitea Secrets Are Good For
### Perfect Use Cases
### Perfect Use Cases
1. **CI/CD Testing** - Run tests with real credentials
2. **Automated Deployment** - Deploy to Proxmox with SSH keys
@@ -22,7 +22,7 @@ Gitea has a **Secrets** feature (like GitHub Actions secrets) that lets you stor
4. **Docker Registry** - Push images with credentials
5. **API Keys** - Access external services during builds
### NOT Good For
### NOT Good For
1. **Runtime secrets** - Your deployed app on Proxmox can't access them
2. **Local development** - Can't use secrets on your laptop
@@ -30,7 +30,7 @@ Gitea has a **Secrets** feature (like GitHub Actions secrets) that lets you stor
---
## 🔧 How to Set Up Gitea Secrets
## How to Set Up Gitea Secrets
### Step 1: Add Secrets to Gitea
@@ -57,7 +57,7 @@ Secrets are accessed with `${{ secrets.SECRET_NAME }}` syntax.
---
## 📝 Example: CI Pipeline with Secrets
## Example: CI Pipeline with Secrets
**File:** `.github/workflows/ci.yml`
@@ -92,11 +92,11 @@ jobs:
--smtp-password "${{ secrets.SMTP_PASSWORD }}"
```
** I've already updated your CI pipeline to use secrets!**
** I've already updated your CI pipeline to use secrets!**
---
## 🚀 Example: Automated Deployment Workflow
## Example: Automated Deployment Workflow
Create `.github/workflows/deploy.yml`:
@@ -152,7 +152,7 @@ jobs:
---
## 🔄 How Secrets Flow to Your Server
## How Secrets Flow to Your Server
### Option 1: Deploy Workflow Updates `.env` (Recommended)
@@ -201,11 +201,11 @@ jobs:
---
## 🎯 Recommended Setup for Your POTE Project
## Recommended Setup for Your POTE Project
### For CI/CD (Testing):
**Use Gitea Secrets**
**Use Gitea Secrets**
```yaml
# .github/workflows/ci.yml (already updated!)
@@ -216,7 +216,7 @@ env:
### For Deployed Server (Proxmox):
**Keep using `.env` file**
**Keep using `.env` file**
Why?
- Simpler for manual SSH access
@@ -227,7 +227,7 @@ Why?
---
## 🚀 Complete Workflow: Gitea → Proxmox
## Complete Workflow: Gitea → Proxmox
### 1. Store Secrets in Gitea
@@ -264,27 +264,27 @@ git push origin main
---
## ⚠️ Important Limitations
## Important Limitations
### Gitea Secrets CAN'T:
Be accessed outside of workflows
Be used in local `python script.py` runs
Be read by cron jobs on Proxmox (directly)
Replace `.env` for runtime application config
Be accessed outside of workflows
Be used in local `python script.py` runs
Be read by cron jobs on Proxmox (directly)
Replace `.env` for runtime application config
### Gitea Secrets CAN:
Secure your CI/CD pipeline
Deploy safely without exposing passwords in git
Update `.env` on server during deployment
Run automated tests with real credentials
Secure your CI/CD pipeline
Deploy safely without exposing passwords in git
Update `.env` on server during deployment
Run automated tests with real credentials
---
## 🔒 Security Best Practices
## Security Best Practices
### DO:
### DO:
1. **Store ALL sensitive data as Gitea secrets**
- SMTP passwords
@@ -300,10 +300,10 @@ git push origin main
3. **Never echo secrets**
```yaml
# ❌ BAD - exposes in logs
# BAD - exposes in logs
- run: echo "${{ secrets.PASSWORD }}"
# ✅ GOOD - masked automatically
# GOOD - masked automatically
- run: use_password "${{ secrets.PASSWORD }}"
```
@@ -311,7 +311,7 @@ git push origin main
- Update in Gitea UI
- Re-run deployment workflow
### DON'T:
### DON'T:
1. **Commit secrets to git** (even private repos)
2. **Share secrets via Slack/email**
@@ -320,18 +320,18 @@ git push origin main
---
## 📊 Comparison: Where to Store Secrets
## Comparison: Where to Store Secrets
| Storage | CI/CD | Deployed App | Easy Updates | Security |
|---------|-------|--------------|--------------|----------|
| **Gitea Secrets** | Perfect | No | Via workflow | ⭐⭐⭐⭐⭐ |
| **`.env` file** | No | Perfect | `nano .env` | ⭐⭐⭐ |
| **Environment Vars** | Yes | Yes | Harder | ⭐⭐⭐⭐ |
| **Both (Recommended)** | Yes | Yes | Automated | ⭐⭐⭐⭐⭐ |
| **Gitea Secrets** | Perfect | No | Via workflow | |
| **`.env` file** | No | Perfect | `nano .env` | |
| **Environment Vars** | Yes | Yes | Harder | |
| **Both (Recommended)** | Yes | Yes | Automated | |
---
## 🎯 My Recommendation for You
## My Recommendation for You
### Use BOTH:
@@ -345,21 +345,21 @@ git push origin main
2. Commit code changes
3. Push to Gitea
4. Workflow runs:
- Tests with Gitea secrets
- Deploys to Proxmox
- Updates .env with secrets
5. Proxmox app reads from .env
- Tests with Gitea secrets
- Deploys to Proxmox
- Updates .env with secrets
5. Proxmox app reads from .env
```
**This gives you:**
- Secure CI/CD
- Easy manual SSH access
- Automated deployments
- No passwords in git
- Secure CI/CD
- Easy manual SSH access
- Automated deployments
- No passwords in git
---
## 🚀 Next Steps
## Next Steps
### 1. Add Secrets to Gitea (5 minutes)
@@ -387,7 +387,7 @@ I can create `.github/workflows/deploy.yml` if you want automated deployments!
---
## 💡 Quick Commands
## Quick Commands
### Add SSH Key to Gitea (for deployment):
@@ -409,12 +409,12 @@ git commit --allow-empty -m "Test secrets"
git push
# Check Gitea Actions tab
# Look for green checkmarks
# Look for green checkmarks
```
---
## 📚 See Also
## See Also
- **[docs/13_secrets_management.md](docs/13_secrets_management.md)** - All secrets options
- **[.github/workflows/ci.yml](.github/workflows/ci.yml)** - Updated with secrets support
@@ -422,16 +422,16 @@ git push
---
## Summary
## Summary
**YES, use Gitea secrets!** They're perfect for:
- CI/CD pipelines
- Automated deployments
- Keeping passwords out of git
- CI/CD pipelines
- Automated deployments
- Keeping passwords out of git
**But ALSO keep `.env` on Proxmox** for:
- Runtime application config
- Manual SSH access
- Cron jobs
- Runtime application config
- Manual SSH access
- Cron jobs
**Best of both worlds:** Gitea secrets deploy and update the `.env` file automatically! 🚀
**Best of both worlds:** Gitea secrets deploy and update the `.env` file automatically!
+4 -4
View File
@@ -117,6 +117,7 @@ make beszel-install-agents BESZEL_ONLY=pote-236 # if agent not yet installed
| P | Task | Owner | Status |
|---|------|-------|--------|
| **P1** | Proxmox backup schedule for LXC **236** on pve10 | @you | ⏳ |
| **P3** | Dedicated `pote@levkine.ca` mailbox (vs shared `alerts@`) | @you | optional / low |
| **P3** | Git deploy on LXC (replace rsync-only) | @agent | optional / low |
| **P3** | Kuma LAN health monitor | @agent | optional / low |
@@ -125,7 +126,6 @@ make beszel-install-agents BESZEL_ONLY=pote-236 # if agent not yet installed
**Closed 2026-07-11:** Beszel agent on pote-236 (`beszel-agent` active).
**Closed 2026-07-12:** `make deploy-pote` synced to LXC 236; 07:00 cron email delivered (`idobkin@gmail.com`, log OK). DB ~1129 trades / 80 officials.
**Closed 2026-07-15:** Proxmox backup — LXC 236 confirmed in pve10 vzdump job vmid list. Daily email re-verified through Jul 15 (44 successes logged); DB now 85 officials / 1161 trades. **Weekly report bug fixed** — caller in `report_generator.py` passed stale kwarg names (`days_lookback`/`min_suspicious_trades`/`min_timing_score`) to `PatternDetector.identify_repeat_offenders(lookback_days, min_suspicious_rate)`; had been failing silently every Sun 08:00 since ≥2026-07-05. Fixed + deployed to `.48` + verified `generate_weekly_summary()`/`format_as_text()` run clean (next real send: Sun 08:00).
### Not planned (unless you want them)
@@ -150,9 +150,9 @@ make beszel-install-agents BESZEL_ONLY=pote-236 # if agent not yet installed
| Doc | Purpose |
|-----|---------|
| [EMAIL_SETUP.md](../EMAIL_SETUP.md) | SMTP / Mailcow / levkine.ca |
| [AUTOMATION_QUICKSTART.md](../AUTOMATION_QUICKSTART.md) | Cron + reports |
| [PROXMOX_QUICKSTART.md](../PROXMOX_QUICKSTART.md) | Original LXC provisioning |
| [EMAIL_SETUP.md](EMAIL_SETUP.md) | SMTP / Mailcow / levkine.ca |
| [AUTOMATION_QUICKSTART.md](AUTOMATION_QUICKSTART.md) | Cron + reports |
| [PROXMOX_QUICKSTART.md](PROXMOX_QUICKSTART.md) | Original LXC provisioning |
| Ansible `docs/guides/projects-handoff-2026-05-26.md` | Multi-project homelab context |
| Ansible `docs/guides/smtp-inventory.md` | Mailboxes |
@@ -1,6 +1,6 @@
# Local Testing Guide for POTE
## Testing Locally Before Deployment
## Testing Locally Before Deployment
### Quick Test - Run Full Suite
@@ -10,18 +10,18 @@ source venv/bin/activate
pytest -v
```
**Expected Result:** All 55 tests should pass
**Expected Result:** All 55 tests should pass
---
## 📊 Current Data Status
## Current Data Status
### Live Data Status: **NOT LIVE YET**
### Live Data Status: **NOT LIVE YET**
**Why?**
- 🔴 **House Stock Watcher API is DOWN** (domain issues, unreachable)
- 🟢 **yfinance works** (for price data)
- 🟡 **Sample data available** (5 trades from fixtures)
- **House Stock Watcher API is DOWN** (domain issues, unreachable)
- **yfinance works** (for price data)
- **Sample data available** (5 trades from fixtures)
### What Data Do You Have?
@@ -40,7 +40,7 @@ This will show:
---
## 🧪 Testing Analytics Locally
## Testing Analytics Locally
### 1. Unit Tests (Fast, No External Dependencies)
@@ -53,10 +53,10 @@ pytest tests/test_analytics_integration.py -v
```
These tests:
- Create synthetic price data
- Simulate trades with known returns
- Verify calculations are correct
- Test edge cases (missing data, sell trades, etc.)
- Create synthetic price data
- Simulate trades with known returns
- Verify calculations are correct
- Test edge cases (missing data, sell trades, etc.)
### 2. Manual Test with Local Database
@@ -92,27 +92,27 @@ calc = ReturnCalculator(session)
---
## 📦 What Gets Tested?
## What Gets Tested?
### Core Functionality (All Working )
### Core Functionality (All Working )
1. **Database Models** - Officials, Securities, Trades, Prices
2. **Data Ingestion** - Trade loading, security enrichment
3. **Analytics Engine** - Returns, benchmarks, metrics
4. **Edge Cases** - Missing data, sell trades, disclosure lags
### Integration Tests Cover:
- Return calculations over multiple time windows (30/60/90/180 days)
- Benchmark comparisons (stock vs SPY/QQQ)
- Abnormal return (alpha) calculations
- Official performance summaries
- Sector analysis
- Disclosure timing analysis
- Top performer rankings
- System-wide statistics
- Return calculations over multiple time windows (30/60/90/180 days)
- Benchmark comparisons (stock vs SPY/QQQ)
- Abnormal return (alpha) calculations
- Official performance summaries
- Sector analysis
- Disclosure timing analysis
- Top performer rankings
- System-wide statistics
---
## 🔄 Getting Live Data
## Getting Live Data
### Option 1: Wait for House Stock Watcher API
The API is currently down. Once it's back up:
@@ -164,7 +164,7 @@ export QUIVER_API_KEY="your_key_here"
---
## 📈 After Adding Data, Fetch Prices
## After Adding Data, Fetch Prices
```bash
# This will fetch prices for all securities in your database
@@ -176,12 +176,12 @@ python scripts/enrich_securities.py
---
## 🎯 Complete Local Test Workflow
## Complete Local Test Workflow
```bash
# 1. Run all tests
pytest -v
# All 55 tests should pass
# All 55 tests should pass
# 2. Check local database
python -c "
@@ -210,7 +210,7 @@ python scripts/calculate_all_returns.py --window 90
---
## 🚀 Deploy to Proxmox
## Deploy to Proxmox
Once local tests pass:
@@ -240,7 +240,7 @@ alembic upgrade head
---
## 🐛 Common Issues
## Common Issues
### "No price data found"
**Fix:** Run `python scripts/fetch_sample_prices.py`
@@ -260,7 +260,7 @@ sudo -u postgres psql -c "\l"
---
## 📊 Test Coverage
## Test Coverage
Run tests with coverage report:
@@ -277,19 +277,19 @@ firefox htmlcov/index.html # View coverage report
---
## Summary
## Summary
**Before Deploying:**
1. Run `pytest -v` - all tests pass
2. Run `make lint` - no errors
3. Test locally with sample data
4. Verify analytics work with synthetic prices
1. Run `pytest -v` - all tests pass
2. Run `make lint` - no errors
3. Test locally with sample data
4. Verify analytics work with synthetic prices
**Getting Live Data:**
- 🔴 House Stock Watcher API is down (external issue)
- 🟢 Manual CSV import works NOW
- 🟢 yfinance for prices works NOW
- 🟡 QuiverQuant available (requires free API key)
- House Stock Watcher API is down (external issue)
- Manual CSV import works NOW
- yfinance for prices works NOW
- QuiverQuant available (requires free API key)
**You can deploy and use the system NOW with:**
- Manual data entry
+16 -16
View File
@@ -1,6 +1,6 @@
# Offline Demo - Works Without Internet!
## Full System Working Without Network Access
## Full System Working Without Network Access
Even though your environment doesn't have external internet, **everything works perfectly** using fixture files.
@@ -10,9 +10,9 @@ Even though your environment doesn't have external internet, **everything works
python scripts/ingest_from_fixtures.py
# Output:
# Officials created/updated: 4
# Securities created/updated: 2
# Trades ingested: 5
# Officials created/updated: 4
# Securities created/updated: 2
# Trades ingested: 5
#
# Database totals:
# Total officials: 4
@@ -33,7 +33,7 @@ python scripts/ingest_from_fixtures.py
- NVDA, MSFT, AAPL, TSLA, GOOGL tickers
2. **Offline Scripts**
- `scripts/ingest_from_fixtures.py` - Ingest sample trades ( works now!)
- `scripts/ingest_from_fixtures.py` - Ingest sample trades ( works now!)
- `scripts/fetch_sample_prices.py` - Would need network (yfinance)
3. **28 Passing Tests** - All use mocks, no network required
@@ -67,14 +67,14 @@ with SessionLocal() as session:
### What You Can Do Offline
**Run all tests**: `make test`
**Ingest fixture data**: `python scripts/ingest_from_fixtures.py`
**Query the database**: Use Python REPL or SQLite browser
**Lint & format**: `make lint format`
**Run migrations**: `make migrate`
**Build analytics** (Phase 2): All math/ML works offline!
**Run all tests**: `make test`
**Ingest fixture data**: `python scripts/ingest_from_fixtures.py`
**Query the database**: Use Python REPL or SQLite browser
**Lint & format**: `make lint format`
**Run migrations**: `make migrate`
**Build analytics** (Phase 2): All math/ML works offline!
**Can't do (needs network)**:
**Can't do (needs network)**:
- Fetch live congressional trades from House Stock Watcher
- Fetch stock prices from yfinance
- (But you can add more fixture files to simulate this!)
@@ -108,9 +108,9 @@ You can expand the fixtures for offline development:
## Summary
**The network error is not a problem!** The entire system is designed to work with:
- Fixtures for development/testing
- Real APIs for production (when network available)
- Same code paths for both
- Fixtures for development/testing
- Real APIs for production (when network available)
- Same code paths for both
This is **by design** - makes development fast and tests reliable! 🚀
This is **by design** - makes development fast and tests reliable!
+6 -6
View File
@@ -223,12 +223,12 @@ print(f"Win Rate: {pelosi_stats['win_rate']:.1%}")
## Success Criteria
- Can calculate returns for any trade + window
- Can compare to S&P 500 benchmark
- Can generate official performance summaries
- All calculations tested and accurate
- Performance data stored efficiently
- Documentation complete
- Can calculate returns for any trade + window
- Can compare to S&P 500 benchmark
- Can generate official performance summaries
- All calculations tested and accurate
- Performance data stored efficiently
- Documentation complete
## Timeline
@@ -1,4 +1,4 @@
# Proxmox Quick Start
# Proxmox Quick Start
**Got Proxmox? Deploy POTE in 5 minutes!**
@@ -17,7 +17,7 @@ su - poteapp
cd pote && source venv/bin/activate
python scripts/ingest_from_fixtures.py
# Done!
# Done!
```
---
@@ -81,8 +81,8 @@ source venv/bin/activate
python scripts/ingest_from_fixtures.py
# Should see:
# Officials created: 4
# Trades ingested: 5
# Officials created: 4
# Trades ingested: 5
```
### 5. Setup Cron Jobs
@@ -96,7 +96,7 @@ crontab -e
15 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/enrich_securities.py >> /home/poteapp/logs/enrich.log 2>&1
```
### 6. Done! 🎉
### 6. Done!
Your POTE instance is now running and will:
- Fetch congressional trades daily at 6 AM
@@ -107,12 +107,12 @@ Your POTE instance is now running and will:
## What You Get
**Full PostgreSQL database**
**Automated daily updates** (via cron)
**Isolated environment** (LXC container)
**Easy backups** (Proxmox snapshots)
**Low resource usage** (~500MB RAM)
**Cost**: Just electricity (~$5-10/mo)
**Full PostgreSQL database**
**Automated daily updates** (via cron)
**Isolated environment** (LXC container)
**Easy backups** (Proxmox snapshots)
**Low resource usage** (~500MB RAM)
**Cost**: Just electricity (~$5-10/mo)
---
@@ -231,13 +231,13 @@ pip install -e .
## Next Steps
1. Container running
2. POTE installed
3. Data ingested
4. ⏭ Setup Proxmox backups (Web UI → Datacenter → Backup)
5. ⏭ Configure static IP (if needed)
6. ⏭ Build Phase 2 analytics
7. ⏭ Add FastAPI dashboard
1. Container running
2. POTE installed
3. Data ingested
4. ⏭ Setup Proxmox backups (Web UI → Datacenter → Backup)
5. ⏭ Configure static IP (if needed)
6. ⏭ Build Phase 2 analytics
7. ⏭ Add FastAPI dashboard
---
@@ -264,10 +264,10 @@ pct restart 100
---
**Your Proxmox = Enterprise infrastructure at hobby prices!** 🚀
**Your Proxmox = Enterprise infrastructure at hobby prices!**
Cost breakdown:
- Cloud VPS: $20/mo
- Your Proxmox: ~$10/mo (power)
- **Savings: $120/year**
- **Savings: $120/year**
+19 -19
View File
@@ -1,15 +1,15 @@
# POTE Quick Start Guide
## 🚀 Your System is Ready!
## Your System is Ready!
**Container IP**: Check with `ip addr show eth0 | grep "inet"`
**Database**: PostgreSQL on port 5432
**Username**: `poteuser`
**Password**: `changeme123` (⚠️ change in production!)
**Password**: `changeme123` ( change in production!)
---
## 📊 How to Use POTE
## How to Use POTE
### Option 1: Command Line (SSH into container)
@@ -51,7 +51,7 @@ with engine.connect() as conn:
---
## 🎯 Common Tasks
## Common Tasks
### 1. Check System Status
@@ -145,7 +145,7 @@ ORDER BY trade_count DESC;
---
## 📈 Example Workflows
## Example Workflows
### Workflow 1: Daily Update
@@ -237,7 +237,7 @@ print(f"Exported {len(df)} trades to trades_export.csv")
---
## 🔧 Maintenance
## Maintenance
### Update POTE Code
@@ -287,7 +287,7 @@ nano ~/pote/.env
---
## 🌐 Access Methods Summary
## Access Methods Summary
| Method | From Where | Command |
|--------|-----------|---------|
@@ -298,22 +298,22 @@ nano ~/pote/.env
---
## 📚 What Data Do You Have?
## What Data Do You Have?
Right now (Phase 1 complete):
- **Congressional trading data** (from House Stock Watcher)
- **Security information** (tickers, names, sectors)
- **Historical prices** (OHLCV data from yfinance)
- **Official profiles** (name, party, chamber, state)
- **Congressional trading data** (from House Stock Watcher)
- **Security information** (tickers, names, sectors)
- **Historical prices** (OHLCV data from yfinance)
- **Official profiles** (name, party, chamber, state)
Coming next (Phase 2):
- 📊 **Abnormal return calculations**
- 🤖 **Behavioral clustering**
- 🚨 **Research signals** (follow_research, avoid_risk, watch)
- **Abnormal return calculations**
- **Behavioral clustering**
- **Research signals** (follow_research, avoid_risk, watch)
---
## 🎓 Learning SQL for POTE
## Learning SQL for POTE
### Count Records
```sql
@@ -349,7 +349,7 @@ GROUP BY o.party;
---
## Troubleshooting
## Troubleshooting
### Can't connect remotely?
```bash
@@ -380,12 +380,12 @@ pip install -e .
---
## 🚀 Next Steps
## Next Steps
1. **Populate with real data**: Run `fetch_congressional_trades.py` regularly
2. **Set up cron job** for automatic daily updates
3. **Build analytics** (Phase 2) - abnormal returns, signals
4. **Create dashboard** (Phase 3) - web interface for exploration
Ready to build Phase 2 analytics? Just ask! 📈
Ready to build Phase 2 analytics? Just ask!
@@ -1,15 +1,15 @@
# 🚀 POTE Quick Setup Card
# POTE Quick Setup Card
## 📍 Your Configuration
## Your Configuration
**Email Server:** `mail.levkin.ca`
**Email Account:** `test@levkin.ca`
**Database:** PostgreSQL (configured)
**Status:** Ready for deployment
**Status:** Ready for deployment
---
## 3-Step Setup
## 3-Step Setup
### Step 1: Add Your Password (30 seconds)
@@ -27,7 +27,7 @@ source venv/bin/activate
python scripts/send_daily_report.py --to test@levkin.ca --test-smtp
```
**Check test@levkin.ca inbox** - you should receive a test email!
**Check test@levkin.ca inbox** - you should receive a test email!
### Step 3: Automate (2 minutes)
@@ -37,13 +37,13 @@ python scripts/send_daily_report.py --to test@levkin.ca --test-smtp
# Choose time: 6 AM (recommended)
```
**Done!** 🎉 You'll now receive:
**Done!** You'll now receive:
- Daily reports at 6 AM
- Weekly reports on Sundays
---
## 📦 Deployment to Proxmox (5 minutes)
## Deployment to Proxmox (5 minutes)
### On Proxmox Host:
@@ -79,7 +79,7 @@ bash scripts/proxmox_setup.sh
---
## 🔍 Quick Commands
## Quick Commands
### On Deployed Server (SSH)
@@ -105,7 +105,7 @@ ls -lh ~/logs/*.txt
---
## 📧 Email Configuration (.env)
## Email Configuration (.env)
```env
SMTP_HOST=mail.levkin.ca
@@ -123,27 +123,27 @@ REPORT_RECIPIENTS=test@levkin.ca,user2@example.com,user3@example.com
---
## 📊 What You'll Receive
## What You'll Receive
### Daily Report (6 AM)
```
New congressional trades
Market alerts (unusual activity)
Suspicious timing detections
Summary statistics
New congressional trades
Market alerts (unusual activity)
Suspicious timing detections
Summary statistics
```
### Weekly Report (Sunday 8 AM)
```
Most active officials
Most traded securities
Repeat offenders
Pattern analysis
Most active officials
Most traded securities
Repeat offenders
Pattern analysis
```
---
## 🔧 Troubleshooting
## Troubleshooting
### Email Not Working?
@@ -191,19 +191,19 @@ tail -50 ~/logs/daily_run.log
---
## 📚 Documentation
## Documentation
| Document | Purpose |
|----------|---------|
| **[EMAIL_SETUP.md](EMAIL_SETUP.md)** | Your levkin.ca setup guide |
| **[DEPLOYMENT_AND_AUTOMATION.md](DEPLOYMENT_AND_AUTOMATION.md)** | Answers all questions |
| **[EMAIL_SETUP.md](EMAIL_SETUP.md)** | Your levkin.ca setup guide |
| **[DEPLOYMENT_AND_AUTOMATION.md](DEPLOYMENT_AND_AUTOMATION.md)** | Answers all questions |
| **[AUTOMATION_QUICKSTART.md](AUTOMATION_QUICKSTART.md)** | Quick automation guide |
| **[PROXMOX_QUICKSTART.md](PROXMOX_QUICKSTART.md)** | Proxmox deployment |
| **[QUICKSTART.md](QUICKSTART.md)** | Usage guide |
---
## Checklist
## Checklist
**Local Development:**
- [ ] `.env` file created with password
@@ -227,18 +227,18 @@ tail -50 ~/logs/daily_run.log
---
## 🎯 Your Current Status
## Your Current Status
**Code:** Complete (93 tests passing)
**Monitoring:** 3-phase system operational
**CI/CD:** Pipeline ready (.github/workflows/ci.yml)
**Email:** Configured for test@levkin.ca
**Code:** Complete (93 tests passing)
**Monitoring:** 3-phase system operational
**CI/CD:** Pipeline ready (.github/workflows/ci.yml)
**Email:** Configured for test@levkin.ca
**Deployment:** Ready to deploy to Proxmox
**Automation:** Ready to set up with `setup_cron.sh`
---
## 🚀 Next Action
## Next Action
**Right now (local testing):**
```bash
@@ -255,9 +255,9 @@ cd ~/pote
./scripts/setup_cron.sh
```
**That's it! 🎉**
**That's it! **
---
**Everything is ready - just add your password and test!** 📧
**Everything is ready - just add your password and test!**
+27 -27
View File
@@ -3,25 +3,25 @@
**Last Updated**: 2025-12-14
**Version**: Phase 1 Complete (PR1 + PR2)
## 🎉 What's Working Now
## What's Working Now
### Data Ingestion (FREE!)
**Congressional Trades**: Live ingestion from House Stock Watcher
**Stock Prices**: Daily OHLCV from yfinance
**Officials**: Auto-populated from trade disclosures
**Securities**: Auto-created, ready for enrichment
**Congressional Trades**: Live ingestion from House Stock Watcher
**Stock Prices**: Daily OHLCV from yfinance
**Officials**: Auto-populated from trade disclosures
**Securities**: Auto-created, ready for enrichment
### Database
**Schema**: Normalized (officials, securities, trades, prices, metrics stubs)
**Migrations**: Alembic configured and applied
**DB**: SQLite for dev, PostgreSQL-ready
**Schema**: Normalized (officials, securities, trades, prices, metrics stubs)
**Migrations**: Alembic configured and applied
**DB**: SQLite for dev, PostgreSQL-ready
### Code Quality
**Tests**: 28 passing (86% coverage)
**Linting**: ruff + mypy all green
**Format**: black applied consistently
**Tests**: 28 passing (86% coverage)
**Linting**: ruff + mypy all green
**Format**: black applied consistently
## 📊 Current Stats
## Current Stats
```bash
# Test Suite
@@ -43,7 +43,7 @@ All free/open-source:
- pytest (testing)
```
## 🚀 Quick Commands
## Quick Commands
### Fetch Live Data (FREE!)
```bash
@@ -75,7 +75,7 @@ make format # Format with black
make migrate # Run Alembic migrations
```
## 🏠 Deployment
## Deployment
**Your Proxmox?** Perfect! See [`docs/08_proxmox_deployment.md`](docs/08_proxmox_deployment.md) for:
- LXC container setup (lightweight, recommended)
@@ -90,7 +90,7 @@ Other options in [`docs/07_deployment.md`](docs/07_deployment.md):
- Railway/Fly.io - $5-15/mo
- AWS/GCP - $20-50/mo
## 📂 Project Structure
## Project Structure
```
pote/
@@ -138,7 +138,7 @@ pote/
└── fetch_sample_prices.py # Live price fetch
```
## 💰 Cost Breakdown
## Cost Breakdown
| Component | Cost | Notes |
|-----------|------|-------|
@@ -154,7 +154,7 @@ Optional paid upgrades (NOT needed):
- Financial Modeling Prep: $15/mo (250 calls/day free tier available)
- PostgreSQL hosting: $7+/mo (only if deploying)
## Completed PRs
## Completed PRs
### PR1: Project Scaffold + Price Loader
- [x] Project structure (`src/`, `tests/`, docs)
@@ -176,7 +176,7 @@ Optional paid upgrades (NOT needed):
**See**: [`docs/PR2_SUMMARY.md`](docs/PR2_SUMMARY.md)
## 📋 Next Steps (Phase 2 - Analytics)
## Next Steps (Phase 2 - Analytics)
### PR3: Security Enrichment
- [ ] Enrich securities table with yfinance (names, sectors, exchanges)
@@ -206,20 +206,20 @@ Optional paid upgrades (NOT needed):
**See**: [`docs/00_mvp.md`](docs/00_mvp.md) for full roadmap
## 🔬 Research-Only Reminder
## Research-Only Reminder
**This tool is for private research and transparency analysis only.**
- Not investment advice
- Not a trading system
- No claims about inside information
- Public data only
- Descriptive analytics
- Research transparency
- Not investment advice
- Not a trading system
- No claims about inside information
- Public data only
- Descriptive analytics
- Research transparency
See [`docs/04_safety_ethics.md`](docs/04_safety_ethics.md) for guardrails.
## 🤝 Contributing
## Contributing
This is a personal research project, but if you want to use it:
@@ -229,7 +229,7 @@ This is a personal research project, but if you want to use it:
4. `python scripts/fetch_congressional_trades.py --days 7`
5. Start exploring!
## 📄 License
## License
MIT License (for research/educational use only)
+39 -39
View File
@@ -1,12 +1,12 @@
# POTE Testing Status Report
**Date:** December 15, 2025
**Status:** All Systems Operational - Ready for Deployment
**Status:** All Systems Operational - Ready for Deployment
---
## 🎯 Test Suite Summary
## Test Suite Summary
### **55 Tests - All Passing **
### **55 Tests - All Passing **
```
Platform: Python 3.13.5, pytest-9.0.2
@@ -18,17 +18,17 @@ Coverage: ~85% overall
| Module | Tests | Status | Coverage |
|--------|-------|--------|----------|
| **Analytics** | 18 tests | PASS | 80% |
| **Models** | 7 tests | PASS | 90% |
| **Ingestion** | 14 tests | PASS | 85% |
| **Price Loader** | 8 tests | PASS | 90% |
| **Security Enricher** | 8 tests | PASS | 85% |
| **Analytics** | 18 tests | PASS | 80% |
| **Models** | 7 tests | PASS | 90% |
| **Ingestion** | 14 tests | PASS | 85% |
| **Price Loader** | 8 tests | PASS | 90% |
| **Security Enricher** | 8 tests | PASS | 85% |
---
## 📊 What's Been Tested?
## What's Been Tested?
### Core Database Operations
### Core Database Operations
- [x] Creating and querying Officials
- [x] Creating and querying Securities
- [x] Creating and querying Trades
@@ -36,7 +36,7 @@ Coverage: ~85% overall
- [x] Unique constraints and relationships
- [x] Database migrations (Alembic)
### Data Ingestion
### Data Ingestion
- [x] House Stock Watcher client (with fixtures)
- [x] Trade loading from JSON
- [x] Security enrichment from yfinance
@@ -44,7 +44,7 @@ Coverage: ~85% overall
- [x] Idempotent operations (no duplicates)
- [x] Error handling for missing/invalid data
### Analytics Engine
### Analytics Engine
- [x] Return calculations (buy trades)
- [x] Return calculations (sell trades)
- [x] Multiple time windows (30/60/90/180 days)
@@ -56,7 +56,7 @@ Coverage: ~85% overall
- [x] Top performer rankings
- [x] System-wide statistics
### Edge Cases
### Edge Cases
- [x] Missing price data handling
- [x] Trades with no exit price yet
- [x] Sell trades (inverted returns)
@@ -67,7 +67,7 @@ Coverage: ~85% overall
---
## 🧪 Test Types
## Test Types
### 1. Unit Tests (Fast, Isolated)
**Location:** `tests/test_*.py` (excluding integration)
@@ -100,7 +100,7 @@ Coverage: ~85% overall
---
## 🔧 How to Run Tests Locally
## How to Run Tests Locally
### Quick Test
```bash
@@ -136,7 +136,7 @@ ptw
---
## 🚨 Known Limitations
## Known Limitations
### 1. External API Dependency
**Issue:** House Stock Watcher API is currently DOWN
@@ -161,7 +161,7 @@ ptw
---
## 📈 Performance Benchmarks
## Performance Benchmarks
### Test Execution Time
- **Full suite:** 1.8 seconds
@@ -178,7 +178,7 @@ ptw
---
## 🎯 Pre-Deployment Checklist
## Pre-Deployment Checklist
### Before Deploying to Proxmox:
@@ -215,7 +215,7 @@ python scripts/enrich_securities.py
---
## 🔄 Continuous Testing
## Continuous Testing
### Git Pre-Commit Hook (Optional)
```bash
@@ -247,7 +247,7 @@ jobs:
---
## 📝 Test Maintenance
## Test Maintenance
### Adding New Tests
@@ -281,36 +281,36 @@ Fixtures are in `tests/conftest.py`:
---
## 🎉 Summary
## Summary
### Current Status: **PRODUCTION READY**
### Current Status: **PRODUCTION READY**
**What Works:**
- All 55 tests passing
- Full analytics pipeline functional
- Database operations solid
- Data ingestion from multiple sources
- Price fetching from yfinance
- Security enrichment
- Return calculations
- Benchmark comparisons
- Performance metrics
- CLI scripts operational
- All 55 tests passing
- Full analytics pipeline functional
- Database operations solid
- Data ingestion from multiple sources
- Price fetching from yfinance
- Security enrichment
- Return calculations
- Benchmark comparisons
- Performance metrics
- CLI scripts operational
**What's Missing:**
- Live congressional trade API (external issue - House Stock Watcher down)
- Live congressional trade API (external issue - House Stock Watcher down)
- **Workaround:** Manual import, CSV, or alternative APIs available
**Next Steps:**
1. Tests are complete
2. Code is ready
3. ➡️ **Deploy to Proxmox** (or continue with Phase 2 features)
4. ➡️ Add more data sources
5. ➡️ Build dashboard (Phase 3)
1. Tests are complete
2. Code is ready
3. **Deploy to Proxmox** (or continue with Phase 2 features)
4. Add more data sources
5. Build dashboard (Phase 3)
---
## 📞 Need Help?
## Need Help?
See:
- `LOCAL_TEST_GUIDE.md` - Detailed local testing instructions
+18 -18
View File
@@ -1,6 +1,6 @@
# POTE Watchlist & Trading Reports
## 🎯 Get Trading Reports 1 Hour Before Market Close
## Get Trading Reports 1 Hour Before Market Close
### Quick Setup
@@ -25,7 +25,7 @@ crontab -e
---
## 📋 Watchlist System
## Watchlist System
### Who's on the Default Watchlist?
@@ -48,7 +48,7 @@ crontab -e
---
## 🔧 Managing Your Watchlist
## Managing Your Watchlist
### View Current Watchlist
@@ -104,7 +104,7 @@ This fetches all 535 members of Congress (100 Senate + 435 House).
---
## 📊 Generating Reports
## Generating Reports
### Manual Report Generation
@@ -135,24 +135,24 @@ python scripts/generate_trading_report.py --format json --output report.json
================================================================================
────────────────────────────────────────────────────────────────────────────────
👤 Nancy Pelosi (D-CA, House)
Nancy Pelosi (D-CA, House)
────────────────────────────────────────────────────────────────────────────────
Side Ticker Company Sector Value Trade Date Filed
-------- ------ -------------------------- ---------- ------------------- ---------- ----------
🟢 BUY NVDA NVIDIA Corporation Technology $15,001 - $50,000 2024-11-15 2024-12-01
🔴 SELL MSFT Microsoft Corporation Technology $50,001 - $100,000 2024-11-20 2024-12-01
BUY NVDA NVIDIA Corporation Technology $15,001 - $50,000 2024-11-15 2024-12-01
SELL MSFT Microsoft Corporation Technology $50,001 - $100,000 2024-11-20 2024-12-01
────────────────────────────────────────────────────────────────────────────────
👤 Tommy Tuberville (R-AL, Senate)
Tommy Tuberville (R-AL, Senate)
────────────────────────────────────────────────────────────────────────────────
Side Ticker Company Sector Value Trade Date Filed
-------- ------ -------------------------- ---------- ------------------- ---------- ----------
🟢 BUY SPY SPDR S&P 500 ETF Financial $100,001 - $250,000 2024-11-18 2024-12-02
🟢 BUY AAPL Apple Inc. Technology $50,001 - $100,000 2024-11-22 2024-12-02
🔴 SELL TSLA Tesla, Inc. Automotive $15,001 - $50,000 2024-11-25 2024-12-02
BUY SPY SPDR S&P 500 ETF Financial $100,001 - $250,000 2024-11-18 2024-12-02
BUY AAPL Apple Inc. Technology $50,001 - $100,000 2024-11-22 2024-12-02
SELL TSLA Tesla, Inc. Automotive $15,001 - $50,000 2024-11-25 2024-12-02
================================================================================
📊 SUMMARY
SUMMARY
================================================================================
Total Trades: 5
@@ -223,7 +223,7 @@ Runs at 8 AM and 3 PM daily (weekdays).
---
## 📧 Email Reports (Optional)
## Email Reports (Optional)
### Setup Email Notifications
@@ -256,7 +256,7 @@ python scripts/send_email.py /tmp/report.html your-email@example.com
---
## 🎯 Typical Workflow
## Typical Workflow
### Daily Routine (3 PM ET)
@@ -282,7 +282,7 @@ python scripts/send_email.py /tmp/report.html your-email@example.com
---
## 🔍 Finding More Officials
## Finding More Officials
### Public Resources
@@ -318,7 +318,7 @@ Add committee members to your watchlist.
---
## 📈 Example Cron Setup
## Example Cron Setup
```bash
# Edit crontab
@@ -341,7 +341,7 @@ This gives you:
---
## 🚀 Quick Start Summary
## Quick Start Summary
```bash
# 1. Create watchlist
@@ -363,7 +363,7 @@ cat reports/trading_report_$(date +%Y%m%d).txt
---
## FAQ
## FAQ
**Q: Why are all the trades old (30-45 days)?**
**A:** Federal law (STOCK Act) gives Congress 30-45 days to file. This is normal.
@@ -1,8 +1,8 @@
# 🎉 POTE Monitoring System - ALL PHASES COMPLETE!
# POTE Monitoring System - ALL PHASES COMPLETE!
## **What Was Built (3 Phases)**
## **What Was Built (3 Phases)**
### **Phase 1: Real-Time Market Monitoring**
### **Phase 1: Real-Time Market Monitoring**
**Detects unusual market activity in congressional tickers**
**Features:**
@@ -20,11 +20,11 @@
- `MarketAlert` model - Database storage
- `monitor_market.py` - CLI tool
**Tests:** 14 passing
**Tests:** 14 passing
---
### **Phase 2: Disclosure Timing Correlation**
### **Phase 2: Disclosure Timing Correlation**
**Matches trades to prior market alerts when disclosures appear**
**Features:**
@@ -50,11 +50,11 @@
- `DisclosureCorrelator` - Correlation engine
- `analyze_disclosure_timing.py` - CLI tool
**Tests:** 13 passing
**Tests:** 13 passing
---
### **Phase 3: Pattern Detection & Rankings**
### **Phase 3: Pattern Detection & Rankings**
**Cross-official analysis and comparative rankings**
**Features:**
@@ -71,19 +71,19 @@
- `PatternDetector` - Pattern analysis engine
- `generate_pattern_report.py` - CLI tool
**Tests:** 11 passing
**Tests:** 11 passing
---
## 📊 **Complete System Architecture**
## **Complete System Architecture**
```
┌─────────────────────────────────────────────────────────────┐
│ PHASE 1: Real-Time Monitoring │
│ ──────────────────────────────────── │
🔔 Monitor congressional tickers
📊 Detect unusual activity
💾 Log alerts to database
│ Monitor congressional tickers │
│ Detect unusual activity │
│ Log alerts to database │
└─────────────────────────────────────────────────────────────┘
[30-45 days pass]
@@ -91,25 +91,25 @@
┌─────────────────────────────────────────────────────────────┐
│ PHASE 2: Disclosure Correlation │
│ ─────────────────────────────── │
📋 New congressional trades filed
🔗 Match to prior alerts
📈 Calculate timing scores
🚩 Flag suspicious trades
│ New congressional trades filed │
│ Match to prior alerts │
│ Calculate timing scores │
│ Flag suspicious trades │
└─────────────────────────────────────────────────────────────┘
┌─────────────────────────────────────────────────────────────┐
│ PHASE 3: Pattern Detection │
│ ────────────────────────── │
📊 Rank officials by timing
🔥 Identify repeat offenders
📈 Compare parties, sectors, tickers
📋 Generate comprehensive reports
│ Rank officials by timing │
│ Identify repeat offenders │
│ Compare parties, sectors, tickers │
│ Generate comprehensive reports │
└─────────────────────────────────────────────────────────────┘
```
---
## 🚀 **Usage Guide**
## **Usage Guide**
### **1. Set Up Monitoring (Run Daily)**
@@ -169,7 +169,7 @@ python scripts/generate_pattern_report.py --days 365 --format json --output patt
---
## 📋 **Example Reports**
## **Example Reports**
### **Timing Analysis Report**
@@ -179,7 +179,7 @@ python scripts/generate_pattern_report.py --days 365 --format json --output patt
3 Trades with Timing Advantages Detected
================================================================================
🚨 #1 - HIGHLY SUSPICIOUS (Timing Score: 85/100)
#1 - HIGHLY SUSPICIOUS (Timing Score: 85/100)
────────────────────────────────────────────────────────────────────────────────
Official: Nancy Pelosi
Ticker: NVDA
@@ -187,16 +187,16 @@ Side: BUY
Trade Date: 2024-01-15
Value: $15,001-$50,000
📊 Timing Analysis:
Timing Analysis:
Prior Alerts: 3
Recent Alerts (7d): 2
High Severity: 2
Avg Severity: 7.5/10
💡 Assessment: Trade occurred after 3 alerts, including 2 high-severity.
Assessment: Trade occurred after 3 alerts, including 2 high-severity.
High likelihood of timing advantage.
🔔 Prior Market Alerts:
Prior Market Alerts:
Timestamp Type Severity Timing
2024-01-12 10:30:00 Unusual Volume 8/10 3 days before
2024-01-13 14:15:00 Price Spike 7/10 2 days before
@@ -211,27 +211,27 @@ Timestamp Type Severity Timing
Period: 365 days
================================================================================
📊 SUMMARY
SUMMARY
────────────────────────────────────────────────────────────────────────────────
Officials Analyzed: 45
Repeat Offenders: 8
Average Timing Score: 42.3/100
🚨 TOP 10 MOST SUSPICIOUS OFFICIALS (By Timing Score)
TOP 10 MOST SUSPICIOUS OFFICIALS (By Timing Score)
================================================================================
Rank Official Party-State Chamber Trades Suspicious Rate Avg Score
──── ─────────────────────── ─────────── ─────── ────── ────────── ────── ─────────
🚨 1 Tommy Tuberville R-AL Senate 47 35/47 74.5% 72.5/100
🚨 2 Nancy Pelosi D-CA House 38 28/38 73.7% 71.2/100
🔴 3 Dan Crenshaw R-TX House 25 15/25 60.0% 65.8/100
🔴 4 Marjorie Taylor Greene R-GA House 19 11/19 57.9% 63.2/100
🟡 5 Josh Gottheimer D-NJ House 31 14/31 45.2% 58.7/100
1 Tommy Tuberville R-AL Senate 47 35/47 74.5% 72.5/100
2 Nancy Pelosi D-CA House 38 28/38 73.7% 71.2/100
3 Dan Crenshaw R-TX House 25 15/25 60.0% 65.8/100
4 Marjorie Taylor Greene R-GA House 19 11/19 57.9% 63.2/100
5 Josh Gottheimer D-NJ House 31 14/31 45.2% 58.7/100
🔥 REPEAT OFFENDERS (50%+ Suspicious Trades)
REPEAT OFFENDERS (50%+ Suspicious Trades)
================================================================================
🚨 Tommy Tuberville (R-AL, Senate)
Tommy Tuberville (R-AL, Senate)
Trades: 47 | Suspicious: 35 (74.5%)
Avg Timing Score: 72.5/100
Pattern: HIGHLY SUSPICIOUS - Majority of trades show timing advantage
@@ -239,9 +239,9 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## 📈 **Test Coverage**
## **Test Coverage**
**Total: 93 tests, all passing **
**Total: 93 tests, all passing **
- **Phase 1 (Monitoring):** 14 tests
- **Phase 2 (Correlation):** 13 tests
@@ -252,7 +252,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## 🎯 **Key Insights the System Provides**
## **Key Insights the System Provides**
### **1. Individual Official Analysis**
- Which officials consistently trade before unusual activity?
@@ -282,7 +282,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## 🔧 **Automated Workflow**
## **Automated Workflow**
### **Daily Routine (Recommended)**
@@ -299,7 +299,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## 📊 **Database Schema**
## **Database Schema**
**New Table: `market_alerts`**
```sql
@@ -315,7 +315,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## 🎓 **Interpretation Guide**
## **Interpretation Guide**
### **Timing Scores**
- **80-100:** Highly suspicious - Multiple high-severity alerts before trade
@@ -337,15 +337,15 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## ⚠️ **Important Disclaimers**
## **Important Disclaimers**
### **Legal & Ethical**
1. All data is public and legally obtained
2. Analysis is retrospective (30-45 day lag)
3. For research and transparency only
4. NOT investment advice
5. NOT proof of illegal activity (requires investigation)
6. Statistical patterns ≠ legal evidence
1. All data is public and legally obtained
2. Analysis is retrospective (30-45 day lag)
3. For research and transparency only
4. NOT investment advice
5. NOT proof of illegal activity (requires investigation)
6. Statistical patterns ≠ legal evidence
### **Technical Limitations**
1. Cannot identify WHO is trading in real-time
@@ -356,7 +356,7 @@ Rank Official Party-State Chamber Trades Suspicious Rate
---
## 🚀 **Deployment Checklist**
## **Deployment Checklist**
### **On Proxmox Container**
@@ -383,7 +383,7 @@ python scripts/generate_pattern_report.py --days 365
---
## 📚 **Documentation**
## **Documentation**
- **`docs/11_live_market_monitoring.md`** - Deep dive into monitoring
- **`LOCAL_TEST_GUIDE.md`** - Testing instructions
@@ -392,23 +392,23 @@ python scripts/generate_pattern_report.py --days 365
---
## 🎉 **Achievement Unlocked!**
## **Achievement Unlocked!**
**You now have a complete system that:**
Monitors real-time market activity
Correlates trades to prior alerts
Calculates timing advantage scores
Identifies repeat offenders
Ranks officials by suspicion
Generates comprehensive reports
93 tests confirming it works
Monitors real-time market activity
Correlates trades to prior alerts
Calculates timing advantage scores
Identifies repeat offenders
Ranks officials by suspicion
Generates comprehensive reports
93 tests confirming it works
**This is a production-ready transparency and research tool!** 🚀
**This is a production-ready transparency and research tool!**
---
## 🔜 **Potential Future Enhancements**
## **Potential Future Enhancements**
### **Phase 4 Ideas (Optional)**
- Email/SMS alerts for high-severity patterns
@@ -420,6 +420,6 @@ python scripts/generate_pattern_report.py --days 365
- Automated PDF reports
- Historical performance tracking
**But the core system is COMPLETE and FUNCTIONAL now!**
**But the core system is COMPLETE and FUNCTIONAL now!**
@@ -1,6 +1,6 @@
# PR1 Summary: Project Scaffold + DB + Price Loader
**Status**: Complete
**Status**: Complete
**Date**: 2025-12-13
## What was built
@@ -36,7 +36,7 @@ Includes proper indexes, unique constraints, and relationships.
- `tests/conftest.py`: fixtures for in-memory DB, sample officials/securities/trades/prices
- `tests/test_models.py`: model creation, relationships, unique constraints, queries (7 tests)
- `tests/test_price_loader.py`: loader logic, idempotency, upsert, mocking yfinance (8 tests)
- **Result**: 15 tests, all passing
- **Result**: 15 tests, all passing
### 6. Tooling
- **Black** + **ruff** configured and run (all code formatted + linted)
@@ -1,6 +1,6 @@
# PR2 Summary: Congressional Trade Ingestion
**Status**: Complete
**Status**: Complete
**Date**: 2025-12-14
## What was built
@@ -26,7 +26,7 @@
- `tests/fixtures/sample_house_watcher.json`: 5 realistic sample transactions
- Includes House + Senate, Democrats + Republicans, various tickers
### 4. Tests (13 new tests, all passing )
### 4. Tests (13 new tests, all passing )
**`tests/test_house_watcher.py` (8 tests)**:
- Amount range parsing (with range, single value, invalid)
- Transaction type normalization
@@ -58,9 +58,9 @@
python scripts/fetch_congressional_trades.py --days 30
# Sample output:
# Officials created/updated: 47
# Securities created/updated: 89
# Trades ingested: 234
# Officials created/updated: 47
# Securities created/updated: 89
# Trades ingested: 234
```
### Database Queries
@@ -1,6 +1,6 @@
# PR3 Summary: Security Enrichment + Deployment
**Status**: Complete
**Status**: Complete
**Date**: 2025-12-14
## What was built
@@ -32,7 +32,7 @@
python scripts/enrich_securities.py --force
```
### 3. Tests (9 new tests, all passing )
### 3. Tests (9 new tests, all passing )
**`tests/test_security_enricher.py`**:
- Successful enrichment with complete data
- ETF detection and classification
@@ -72,7 +72,7 @@ python scripts/enrich_securities.py
# Enriched AAPL: Apple Inc. (Technology)
# Enriched TSLA: Tesla, Inc. (Consumer Cyclical)
# Enriched GOOGL: Alphabet Inc. (Communication Services)
# Successfully enriched: 5
# Successfully enriched: 5
```
### Query Enriched Data
@@ -159,10 +159,10 @@ python scripts/update_all_prices.py # To be built in PR4
| Option | Complexity | Cost/month | Best For |
|--------|-----------|------------|----------|
| **Local** | | $0 | Development |
| **VPS + Docker** | ⭐⭐ | $10-20 | Personal deployment |
| **Railway/Fly.io** | | $5-15 | Easy cloud |
| **AWS** | ⭐⭐⭐ | $20-50 | Scalable production |
| **Local** | | $0 | Development |
| **VPS + Docker** | | $10-20 | Personal deployment |
| **Railway/Fly.io** | | $5-15 | Easy cloud |
| **AWS** | | $20-50 | Scalable production |
See [`docs/07_deployment.md`](07_deployment.md) for detailed guides.
@@ -1,6 +1,6 @@
# PR4 Summary: Phase 2 Analytics Foundation
## Completed
## Completed
**Date**: December 15, 2025
**Status**: Complete
@@ -78,14 +78,14 @@ python scripts/calculate_all_returns.py --window 90 --benchmark SPY --top 10
### 3. Tests (`tests/test_analytics.py`)
- Return calculator with sample data
- Buy vs sell trade handling
- Missing data edge cases
- Benchmark comparisons
- Official performance metrics
- Multiple time windows
- Sector analysis
- Timing analysis
- Return calculator with sample data
- Buy vs sell trade handling
- Missing data edge cases
- Benchmark comparisons
- Official performance metrics
- Multiple time windows
- Sector analysis
- Timing analysis
**Test Coverage**: Analytics module fully tested
@@ -288,15 +288,15 @@ with next(get_session()) as session:
- Trades near policy events
- Unusual timing flags
## Success Criteria
## Success Criteria
- Can calculate returns for any trade + window
- Can compare to S&P 500 benchmark
- Can generate official performance summaries
- All calculations tested and accurate
- Performance data calculated on-the-fly
- Documentation complete
- Command-line tools working
- Can calculate returns for any trade + window
- Can compare to S&P 500 benchmark
- Can generate official performance summaries
- All calculations tested and accurate
- Performance data calculated on-the-fly
- Documentation complete
- Command-line tools working
## Testing
@@ -309,7 +309,7 @@ All analytics tests should pass (may have warnings if no price data).
---
**Phase 2 Analytics Foundation: COMPLETE**
**Phase 2 Analytics Foundation: COMPLETE**
**Ready for**: PR5 (Signals), PR6 (API), PR7 (Dashboard)
+3
View File
@@ -0,0 +1,3 @@
# Archive
PR summaries and one-shot status writeups. Prefer the numbered guides and QUICKSTART files.
+2
View File
@@ -41,6 +41,8 @@ where = ["src"]
[tool.ruff]
line-length = 100
target-version = "py311"
[tool.ruff.lint]
select = ["E", "F", "W", "I", "N", "UP", "B", "A", "C4", "SIM", "RET"]
ignore = ["E501"] # Line too long (handled by black)
+46
View File
@@ -0,0 +1,46 @@
#!/usr/bin/env bash
# One-time installer for the local pre-commit gitleaks hook.
# Run once per clone: bash scripts/git-hooks/install.sh
#
# Respects `core.hooksPath` if you've set one (local or global) — some setups
# point git at a hooks dir outside `.git/hooks/` (e.g. a machine-wide
# `~/.git-hooks/`), and installing to `.git/hooks/` in that case would be a
# silent no-op. If an existing hook is already at that path, this chains to
# it so nothing already relying on it breaks.
set -euo pipefail
REPO_ROOT="$(git rev-parse --show-toplevel)"
cd "$REPO_ROOT"
HOOKS_DIR="$(git config --get core.hooksPath || true)"
if [ -z "$HOOKS_DIR" ]; then
HOOKS_DIR=".git/hooks"
elif [[ "$HOOKS_DIR" != /* ]]; then
HOOKS_DIR="$REPO_ROOT/$HOOKS_DIR"
fi
mkdir -p "$HOOKS_DIR"
TARGET="$HOOKS_DIR/pre-commit"
if [ -f "$TARGET" ] && ! grep -q "gitleaks" "$TARGET" 2>/dev/null; then
echo "⚠️ Existing pre-commit hook found at $TARGET that isn't ours — chaining instead of overwriting."
CHAINED="$HOOKS_DIR/pre-commit.d-gitleaks"
cp scripts/git-hooks/pre-commit "$CHAINED"
chmod +x "$CHAINED"
if ! grep -q "pre-commit.d-gitleaks" "$TARGET" 2>/dev/null; then
printf '\n# Added by levkinops ansible repo (scripts/git-hooks/install.sh)\n"%s"\n' "$CHAINED" >> "$TARGET"
fi
else
cp scripts/git-hooks/pre-commit "$TARGET"
chmod +x "$TARGET"
fi
echo "✓ Installed pre-commit gitleaks hook → $TARGET"
if [ "$HOOKS_DIR" != ".git/hooks" ] && [ "$HOOKS_DIR" != "$REPO_ROOT/.git/hooks" ]; then
echo " (using core.hooksPath=$HOOKS_DIR — applies to every repo that shares this hooksPath)"
fi
if ! command -v gitleaks >/dev/null 2>&1; then
echo " Note: gitleaks isn't installed locally yet. Install it for the hook to actually run:"
echo " macOS: brew install gitleaks"
echo " Linux: see https://github.com/gitleaks/gitleaks#installing"
echo " Until then this hook is a no-op locally (CI still scans every push)."
fi
+27
View File
@@ -0,0 +1,27 @@
#!/usr/bin/env bash
# Local pre-commit secret scan — mirrors the `secret-scanning` CI job (gitleaks)
# so leaked secrets are caught before they ever leave your machine, not just at
# CI time. Installed via `make install-git-hooks`.
#
# Uses the same .gitleaks.toml allowlist as CI. Only scans staged content.
set -euo pipefail
REPO_ROOT="$(git rev-parse --show-toplevel)"
cd "$REPO_ROOT"
if ! command -v gitleaks >/dev/null 2>&1; then
echo "⚠️ gitleaks not installed locally — skipping local secret scan."
echo " Install: brew install gitleaks (CI will still catch it either way)"
exit 0
fi
echo "🔐 Running gitleaks on staged changes..."
if ! gitleaks protect --staged --config .gitleaks.toml --no-banner --redact; then
echo ""
echo "❌ gitleaks found a potential secret in your staged changes."
echo " Fix it, or if it's a false positive, add an allowlist entry to .gitleaks.toml."
echo " Bypass (not recommended): git commit --no-verify"
exit 1
fi
echo "✓ gitleaks: no secrets found in staged changes"
+1 -1
View File
@@ -74,7 +74,7 @@ def main(tickers, interval, once, min_severity, save_report, lookback):
if filtered:
# Generate report
report = alert_mgr.generate_summary_report(filtered, format="text")
report = alert_mgr.generate_summary_report(filtered, output_format="text")
print("\n" + report)
# Save report if requested
+1 -3
View File
@@ -2,14 +2,12 @@
Analytics module for calculating returns, performance metrics, and signals.
"""
from .returns import ReturnCalculator
from .benchmarks import BenchmarkComparison
from .metrics import PerformanceMetrics
from .returns import ReturnCalculator
__all__ = [
"ReturnCalculator",
"BenchmarkComparison",
"PerformanceMetrics",
]
+2 -5
View File
@@ -3,7 +3,7 @@ Benchmark comparison for calculating abnormal returns (alpha).
"""
import logging
from datetime import date, timedelta
from datetime import date
from decimal import Decimal
from sqlalchemy.orm import Session
@@ -60,8 +60,7 @@ class BenchmarkComparison:
return None
# Calculate return
return_pct = ((end_price - start_price) / start_price) * 100
return return_pct
return ((end_price - start_price) / start_price) * 100
def calculate_abnormal_return(
self,
@@ -219,5 +218,3 @@ class BenchmarkComparison:
"benchmark": self.BENCHMARKS.get(benchmark, benchmark),
"window_days": window_days,
}
+10 -29
View File
@@ -4,7 +4,6 @@ Performance metrics and aggregations.
import logging
from collections import defaultdict
from datetime import date
from sqlalchemy import func
from sqlalchemy.orm import Session
@@ -52,11 +51,7 @@ class PerformanceMetrics:
if not official:
return {"error": "Official not found"}
trades = (
self.session.query(Trade)
.filter(Trade.official_id == official_id)
.all()
)
trades = self.session.query(Trade).filter(Trade.official_id == official_id).all()
if not trades:
return {
@@ -70,9 +65,7 @@ class PerformanceMetrics:
# Calculate returns for all trades
returns_data = []
for trade in trades:
result = self.benchmark.compare_trade_to_benchmark(
trade, window_days, benchmark
)
result = self.benchmark.compare_trade_to_benchmark(trade, window_days, benchmark)
if result:
returns_data.append(result)
@@ -96,9 +89,7 @@ class PerformanceMetrics:
worst_trade = min(returns_data, key=lambda x: x["trade_return"])
# Total value traded
total_value = sum(
float(t.value_min or 0) for t in trades if t.value_min
)
total_value = sum(float(t.value_min or 0) for t in trades if t.value_min)
return {
"name": official.name,
@@ -154,20 +145,14 @@ class PerformanceMetrics:
List of sector performance dictionaries
"""
# Get all trades with security info
trades = (
self.session.query(Trade)
.join(Security)
.all()
)
trades = self.session.query(Trade).join(Security).all()
# Group by sector
sector_data = defaultdict(list)
for trade in trades:
sector = trade.security.sector or "Unknown"
result = self.benchmark.compare_trade_to_benchmark(
trade, window_days, benchmark
)
result = self.benchmark.compare_trade_to_benchmark(trade, window_days, benchmark)
if result:
sector_data[sector].append(result)
@@ -180,14 +165,16 @@ class PerformanceMetrics:
returns = [d["trade_return"] for d in data]
alphas = [d["abnormal_return"] for d in data]
results.append({
results.append(
{
"sector": sector,
"trade_count": len(data),
"avg_return": sum(returns) / len(returns),
"avg_alpha": sum(alphas) / len(alphas),
"win_rate": sum(1 for r in returns if r > 0) / len(returns),
"beat_market_rate": sum(1 for a in alphas if a > 0) / len(alphas),
})
}
)
# Sort by average alpha
results.sort(key=lambda x: x["avg_alpha"], reverse=True)
@@ -229,11 +216,7 @@ class PerformanceMetrics:
Returns:
Dictionary with timing statistics
"""
trades = (
self.session.query(Trade)
.filter(Trade.filing_date.isnot(None))
.all()
)
trades = self.session.query(Trade).filter(Trade.filing_date.isnot(None)).all()
if not trades:
return {"error": "No trades with disclosure dates"}
@@ -288,5 +271,3 @@ class PerformanceMetrics:
"benchmark": benchmark,
**aggregate,
}
+8 -6
View File
@@ -94,18 +94,20 @@ class ReturnCalculator:
def calculate_multiple_windows(
self,
trade: Trade,
windows: list[int] = [30, 60, 90, 180],
windows: list[int] | None = None,
) -> dict[int, dict]:
"""
Calculate returns for multiple time windows.
Args:
trade: Trade object
windows: List of window sizes in days
windows: List of window sizes in days (defaults to 30/60/90/180)
Returns:
Dictionary mapping window_days to return metrics
"""
if windows is None:
windows = [30, 60, 90, 180]
results = {}
for window in windows:
result = self.calculate_trade_return(trade, window)
@@ -223,12 +225,13 @@ class ReturnCalculator:
if not prices:
return pd.DataFrame()
# open/high/low are nullable columns; use NaN (pandas-native) when absent.
data = [
{
"date": p.date,
"open": float(p.open),
"high": float(p.high),
"low": float(p.low),
"open": float(p.open) if p.open is not None else float("nan"),
"high": float(p.high) if p.high is not None else float("nan"),
"low": float(p.low) if p.low is not None else float("nan"),
"close": float(p.close),
"volume": p.volume,
}
@@ -236,4 +239,3 @@ class ReturnCalculator:
]
return pd.DataFrame(data)
+16 -32
View File
@@ -3,17 +3,17 @@ SQLAlchemy ORM models for POTE.
Matches the schema defined in docs/02_data_model.md.
"""
from datetime import date, datetime, timezone
from datetime import UTC, date, datetime
from decimal import Decimal
from sqlalchemy import (
DECIMAL,
JSON,
Date,
DateTime,
ForeignKey,
Index,
Integer,
JSON,
String,
Text,
UniqueConstraint,
@@ -35,13 +35,11 @@ class Official(Base):
state: Mapped[str | None] = mapped_column(String(2))
bioguide_id: Mapped[str | None] = mapped_column(String(20), unique=True)
external_ids: Mapped[str | None] = mapped_column(Text) # JSON blob for other IDs
created_at: Mapped[datetime] = mapped_column(
DateTime, default=lambda: datetime.now(timezone.utc)
)
created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(
DateTime,
default=lambda: datetime.now(timezone.utc),
onupdate=lambda: datetime.now(timezone.utc),
default=lambda: datetime.now(UTC),
onupdate=lambda: datetime.now(UTC),
)
# Relationships
@@ -63,13 +61,11 @@ class Security(Base):
sector: Mapped[str | None] = mapped_column(String(100))
industry: Mapped[str | None] = mapped_column(String(100))
asset_type: Mapped[str] = mapped_column(String(50), default="stock") # stock, bond, etc.
created_at: Mapped[datetime] = mapped_column(
DateTime, default=lambda: datetime.now(timezone.utc)
)
created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(
DateTime,
default=lambda: datetime.now(timezone.utc),
onupdate=lambda: datetime.now(timezone.utc),
default=lambda: datetime.now(UTC),
onupdate=lambda: datetime.now(UTC),
)
# Relationships
@@ -107,13 +103,11 @@ class Trade(Base):
# Quality flags (JSON or enum list)
quality_flags: Mapped[str | None] = mapped_column(Text) # e.g., "range_only,delayed_filing"
created_at: Mapped[datetime] = mapped_column(
DateTime, default=lambda: datetime.now(timezone.utc)
)
created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
updated_at: Mapped[datetime] = mapped_column(
DateTime,
default=lambda: datetime.now(timezone.utc),
onupdate=lambda: datetime.now(timezone.utc),
default=lambda: datetime.now(UTC),
onupdate=lambda: datetime.now(UTC),
)
# Relationships
@@ -156,9 +150,7 @@ class Price(Base):
adjusted_close: Mapped[Decimal | None] = mapped_column(DECIMAL(15, 4))
source: Mapped[str] = mapped_column(String(50), default="yfinance")
created_at: Mapped[datetime] = mapped_column(
DateTime, default=lambda: datetime.now(timezone.utc)
)
created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
# Relationships
security: Mapped["Security"] = relationship("Security", back_populates="prices")
@@ -188,9 +180,7 @@ class MetricOfficial(Base):
avg_abnormal_return_1m: Mapped[Decimal | None] = mapped_column(DECIMAL(10, 6))
cluster_label: Mapped[str | None] = mapped_column(String(50))
created_at: Mapped[datetime] = mapped_column(
DateTime, default=lambda: datetime.now(timezone.utc)
)
created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
__table_args__ = (
UniqueConstraint("official_id", "calc_date", "calc_version", name="uq_metrics_official"),
@@ -212,9 +202,7 @@ class MetricTrade(Base):
abnormal_return_1m: Mapped[Decimal | None] = mapped_column(DECIMAL(10, 6))
signal_flags: Mapped[str | None] = mapped_column(Text) # JSON list
created_at: Mapped[datetime] = mapped_column(
DateTime, default=lambda: datetime.now(timezone.utc)
)
created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
__table_args__ = (
UniqueConstraint("trade_id", "calc_date", "calc_version", name="uq_metrics_trade"),
@@ -242,18 +230,14 @@ class MarketAlert(Base):
# Metrics at time of alert
price: Mapped[Decimal | None] = mapped_column(DECIMAL(15, 4))
volume: Mapped[int | None] = mapped_column(Integer)
change_pct: Mapped[Decimal | None] = mapped_column(
DECIMAL(10, 4)
) # Price change %
change_pct: Mapped[Decimal | None] = mapped_column(DECIMAL(10, 4)) # Price change %
# Severity scoring
severity: Mapped[int | None] = mapped_column(Integer) # 1-10 scale
# Metadata
source: Mapped[str] = mapped_column(String(50), default="market_monitor")
created_at: Mapped[datetime] = mapped_column(
DateTime, default=lambda: datetime.now(timezone.utc)
)
created_at: Mapped[datetime] = mapped_column(DateTime, default=lambda: datetime.now(UTC))
# Indexes for efficient queries
__table_args__ = (
+3 -3
View File
@@ -14,6 +14,7 @@ import httpx
logger = logging.getLogger(__name__)
def _default_data_urls() -> tuple[str, ...]:
override = os.environ.get("POTE_HOUSE_DATA_URL", "").strip()
if override:
@@ -235,10 +236,9 @@ def normalize_transaction_type(txn_type: str) -> str:
if "purchase" in txn_lower or "buy" in txn_lower:
return "buy"
elif "sale" in txn_lower or "sell" in txn_lower:
if "sale" in txn_lower or "sell" in txn_lower:
return "sell"
elif "exchange" in txn_lower:
if "exchange" in txn_lower:
return "exchange"
else:
# Default to the original, lowercased
return txn_lower
+2 -2
View File
@@ -4,7 +4,7 @@ Fetches daily OHLCV data for securities and stores in the prices table.
"""
import logging
from datetime import date, datetime, timedelta, timezone
from datetime import UTC, date, datetime, timedelta
from decimal import Decimal
import pandas as pd
@@ -158,7 +158,7 @@ class PriceLoader:
"volume": int(row["volume"]) if pd.notna(row.get("volume")) else None,
"adjusted_close": None, # We'll compute this later if needed
"source": "yfinance",
"created_at": datetime.now(timezone.utc),
"created_at": datetime.now(UTC),
}
records.append(record)
-1
View File
@@ -9,4 +9,3 @@ from .market_monitor import MarketMonitor
from .pattern_detector import PatternDetector
__all__ = ["MarketMonitor", "AlertManager", "DisclosureCorrelator", "PatternDetector"]
+18 -23
View File
@@ -4,8 +4,7 @@ Handles alert filtering, formatting, and delivery.
"""
import logging
from datetime import datetime, timezone
from typing import Any
from datetime import UTC, datetime
from sqlalchemy.orm import Session
@@ -75,21 +74,24 @@ class AlertManager:
Returns:
HTML formatted alert
"""
severity_class = "high" if (alert.severity or 0) >= 7 else "medium" if (alert.severity or 0) >= 4 else "low"
severity_class = (
"high"
if (alert.severity or 0) >= 7
else "medium" if (alert.severity or 0) >= 4 else "low"
)
html = f"""
return f"""
<div class="alert {severity_class}">
<h3>{alert.ticker} - {alert.alert_type.replace('_', ' ').title()}</h3>
<p class="timestamp">{alert.timestamp.strftime('%Y-%m-%d %H:%M:%S')}</p>
<p class="severity">Severity: {alert.severity}/10</p>
<div class="metrics">
<span>Price: ${float(alert.price):.2f}</span>
<span>Price: ${float(alert.price or 0):.2f}</span>
<span>Volume: {alert.volume:,}</span>
<span>Change: {float(alert.change_pct):+.2f}%</span>
<span>Change: {float(alert.change_pct or 0):+.2f}%</span>
</div>
</div>
"""
return html
def filter_alerts(
self,
@@ -117,7 +119,7 @@ class AlertManager:
# Filter by ticker
if tickers:
ticker_set = set(t.upper() for t in tickers)
ticker_set = {t.upper() for t in tickers}
filtered = [a for a in filtered if a.ticker.upper() in ticker_set]
# Filter by alert type
@@ -128,21 +130,20 @@ class AlertManager:
return filtered
def generate_summary_report(
self, alerts: list[MarketAlert], format: str = "text"
self, alerts: list[MarketAlert], output_format: str = "text"
) -> str:
"""
Generate summary report of alerts.
Args:
alerts: List of alerts
format: Output format ('text' or 'html')
output_format: Output format ('text' or 'html')
Returns:
Formatted summary report
"""
if format == "html":
if output_format == "html":
return self._generate_html_summary(alerts)
else:
return self._generate_text_summary(alerts)
def _generate_text_summary(self, alerts: list[MarketAlert]) -> str:
@@ -152,7 +153,7 @@ class AlertManager:
lines = [
"=" * 80,
f" MARKET ACTIVITY ALERTS - {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S')} UTC",
f" MARKET ACTIVITY ALERTS - {datetime.now(UTC).strftime('%Y-%m-%d %H:%M:%S')} UTC",
f" {len(alerts)} Alerts",
"=" * 80,
"",
@@ -180,9 +181,7 @@ class AlertManager:
lines.append(f"🎯 {ticker} - {len(ticker_alerts)} alerts (Max Severity: {max_sev}/10)")
lines.append("" * 80)
for alert in sorted(
ticker_alerts, key=lambda a: a.severity or 0, reverse=True
):
for alert in sorted(ticker_alerts, key=lambda a: a.severity or 0, reverse=True):
lines.append("")
lines.append(self.format_alert_text(alert))
@@ -202,9 +201,7 @@ class AlertManager:
type_counts[alert.alert_type] = type_counts.get(alert.alert_type, 0) + 1
lines.append("\nAlert Types:")
for alert_type, count in sorted(
type_counts.items(), key=lambda x: x[1], reverse=True
):
for alert_type, count in sorted(type_counts.items(), key=lambda x: x[1], reverse=True):
lines.append(f" {alert_type.replace('_', ' ').title():20s}: {count}")
# Top severity alerts
@@ -232,8 +229,8 @@ class AlertManager:
".timestamp { color: #666; font-size: 0.9em; }",
".metrics span { margin-right: 20px; }",
"</style></head><body>",
f"<h1>Market Activity Alerts</h1>",
f"<p><strong>{len(alerts)} Alerts</strong> | {datetime.now(timezone.utc).strftime('%Y-%m-%d %H:%M:%S')} UTC</p>",
"<h1>Market Activity Alerts</h1>",
f"<p><strong>{len(alerts)} Alerts</strong> | {datetime.now(UTC).strftime('%Y-%m-%d %H:%M:%S')} UTC</p>",
]
for alert in sorted(alerts, key=lambda a: a.severity or 0, reverse=True):
@@ -241,5 +238,3 @@ class AlertManager:
html_parts.append("</body></html>")
return "\n".join(html_parts)
+18 -50
View File
@@ -5,14 +5,13 @@ Calculates timing advantage and suspicious activity scores.
"""
import logging
from datetime import date, timedelta, timezone
from decimal import Decimal
from datetime import UTC, date, timedelta
from typing import Any
from sqlalchemy import and_, func
from sqlalchemy import and_
from sqlalchemy.orm import Session
from pote.db.models import MarketAlert, Official, Security, Trade
from pote.db.models import MarketAlert, Security, Trade
logger = logging.getLogger(__name__)
@@ -27,9 +26,7 @@ class DisclosureCorrelator:
"""Initialize disclosure correlator."""
self.session = session
def get_alerts_before_trade(
self, trade: Trade, lookback_days: int = 30
) -> list[MarketAlert]:
def get_alerts_before_trade(self, trade: Trade, lookback_days: int = 30) -> list[MarketAlert]:
"""
Get market alerts that occurred BEFORE a trade.
@@ -50,14 +47,10 @@ class DisclosureCorrelator:
# Convert dates to datetime for comparison
from datetime import datetime
start_dt = datetime.combine(start_date, datetime.min.time()).replace(
tzinfo=timezone.utc
)
end_dt = datetime.combine(end_date, datetime.max.time()).replace(
tzinfo=timezone.utc
)
start_dt = datetime.combine(start_date, datetime.min.time()).replace(tzinfo=UTC)
end_dt = datetime.combine(end_date, datetime.max.time()).replace(tzinfo=UTC)
alerts = (
return (
self.session.query(MarketAlert)
.filter(
and_(
@@ -70,8 +63,6 @@ class DisclosureCorrelator:
.all()
)
return alerts
def calculate_timing_score(
self, trade: Trade, prior_alerts: list[MarketAlert]
) -> dict[str, Any]:
@@ -131,8 +122,7 @@ class DisclosureCorrelator:
)
elif suspicious:
reason = (
f"Trade occurred after {len(prior_alerts)} alerts. "
f"Possible timing advantage."
f"Trade occurred after {len(prior_alerts)} alerts. " f"Possible timing advantage."
)
else:
reason = (
@@ -170,35 +160,27 @@ class DisclosureCorrelator:
timing_analysis = self.calculate_timing_score(trade, prior_alerts)
# Build full analysis
analysis = {
return {
"trade_id": trade.id,
"official_name": trade.official.name if trade.official else None,
"ticker": trade.security.ticker if trade.security else None,
"side": trade.side,
"transaction_date": str(trade.transaction_date),
"filing_date": str(trade.filing_date) if trade.filing_date else None,
"value_range": f"${float(trade.value_min):,.0f}"
+ (
f"-${float(trade.value_max):,.0f}"
if trade.value_max
else "+"
),
"value_range": f"${float(trade.value_min or 0):,.0f}"
+ (f"-${float(trade.value_max):,.0f}" if trade.value_max else "+"),
**timing_analysis,
"prior_alerts": [
{
"timestamp": str(alert.timestamp),
"alert_type": alert.alert_type,
"severity": alert.severity,
"days_before_trade": (
trade.transaction_date - alert.timestamp.date()
).days,
"days_before_trade": (trade.transaction_date - alert.timestamp.date()).days,
}
for alert in prior_alerts
],
}
return analysis
def analyze_recent_disclosures(
self, days: int = 7, min_timing_score: float = 50
) -> list[dict[str, Any]]:
@@ -237,9 +219,7 @@ class DisclosureCorrelator:
f"Found {len(suspicious_trades)} trades with timing score >= {min_timing_score}"
)
return sorted(
suspicious_trades, key=lambda x: x["timing_score"], reverse=True
)
return sorted(suspicious_trades, key=lambda x: x["timing_score"], reverse=True)
def get_official_timing_pattern(
self, official_id: int, lookback_days: int = 365
@@ -258,9 +238,7 @@ class DisclosureCorrelator:
trades = (
self.session.query(Trade)
.filter(
and_(Trade.official_id == official_id, Trade.transaction_date >= since_date)
)
.filter(and_(Trade.official_id == official_id, Trade.transaction_date >= since_date))
.join(Trade.security)
.all()
)
@@ -285,9 +263,7 @@ class DisclosureCorrelator:
highly_suspicious = sum(1 for a in analyses if a.get("highly_suspicious", False))
avg_timing_score = (
sum(a["timing_score"] for a in analyses) / total_trades
if total_trades > 0
else 0
sum(a["timing_score"] for a in analyses) / total_trades if total_trades > 0 else 0
)
# Determine pattern
@@ -311,9 +287,7 @@ class DisclosureCorrelator:
"analyses": analyses,
}
def get_ticker_timing_analysis(
self, ticker: str, lookback_days: int = 365
) -> dict[str, Any]:
def get_ticker_timing_analysis(self, ticker: str, lookback_days: int = 365) -> dict[str, Any]:
"""
Analyze timing patterns for a specific ticker.
@@ -329,9 +303,7 @@ class DisclosureCorrelator:
trades = (
self.session.query(Trade)
.join(Trade.security)
.filter(
and_(Security.ticker == ticker, Trade.transaction_date >= since_date)
)
.filter(and_(Security.ticker == ticker, Trade.transaction_date >= since_date))
.join(Trade.official)
.all()
)
@@ -350,10 +322,6 @@ class DisclosureCorrelator:
"trade_count": len(analyses),
"trades_with_alerts": sum(1 for a in analyses if a["alert_count"] > 0),
"suspicious_count": sum(1 for a in analyses if a["suspicious"]),
"avg_timing_score": round(
sum(a["timing_score"] for a in analyses) / len(analyses), 2
),
"avg_timing_score": round(sum(a["timing_score"] for a in analyses) / len(analyses), 2),
"analyses": sorted(analyses, key=lambda x: x["timing_score"], reverse=True),
}
+9 -15
View File
@@ -4,7 +4,7 @@ Detects unusual activity: volume spikes, price movements, volatility.
"""
import logging
from datetime import datetime, timedelta, timezone
from datetime import UTC, datetime, timedelta
from decimal import Decimal
from typing import Any
@@ -59,7 +59,7 @@ class MarketMonitor:
Returns:
List of alerts detected
"""
alerts = []
alerts: list[dict[str, Any]] = []
try:
stock = yf.Ticker(ticker)
@@ -90,7 +90,7 @@ class MarketMonitor:
{
"ticker": ticker,
"alert_type": "unusual_volume",
"timestamp": datetime.now(timezone.utc),
"timestamp": datetime.now(UTC),
"details": {
"current_volume": int(current_volume),
"avg_volume": int(avg_volume),
@@ -109,10 +109,8 @@ class MarketMonitor:
alerts.append(
{
"ticker": ticker,
"alert_type": "price_spike"
if price_change > 0
else "price_drop",
"timestamp": datetime.now(timezone.utc),
"alert_type": "price_spike" if price_change > 0 else "price_drop",
"timestamp": datetime.now(UTC),
"details": {
"current_price": float(current_price),
"prev_price": float(prev["Close"]),
@@ -129,14 +127,12 @@ class MarketMonitor:
if len(hist) >= 5:
recent_volatility = hist["Close"].iloc[-5:].pct_change().abs().mean()
if recent_volatility > avg_price_change * 2 and avg_price_change > 0:
severity = min(
10, int((recent_volatility / avg_price_change) - 1)
)
severity = min(10, int((recent_volatility / avg_price_change) - 1))
alerts.append(
{
"ticker": ticker,
"alert_type": "high_volatility",
"timestamp": datetime.now(timezone.utc),
"timestamp": datetime.now(UTC),
"details": {
"recent_volatility": round(recent_volatility * 100, 2),
"avg_volatility": round(avg_price_change * 100, 2),
@@ -227,7 +223,7 @@ class MarketMonitor:
Returns:
List of MarketAlert objects
"""
since = datetime.now(timezone.utc) - timedelta(days=days)
since = datetime.now(UTC) - timedelta(days=days)
query = self.session.query(MarketAlert).filter(MarketAlert.timestamp >= since)
@@ -252,7 +248,7 @@ class MarketMonitor:
Returns:
Dict mapping ticker to alert summary
"""
since = datetime.now(timezone.utc) - timedelta(days=days)
since = datetime.now(UTC) - timedelta(days=days)
from sqlalchemy import func
@@ -278,5 +274,3 @@ class MarketMonitor:
}
return summary
+12 -29
View File
@@ -5,13 +5,12 @@ Identifies recurring suspicious behavior and trading patterns.
import logging
from datetime import date, timedelta
from decimal import Decimal
from typing import Any
from sqlalchemy import and_, func
from sqlalchemy import func
from sqlalchemy.orm import Session
from pote.db.models import MarketAlert, Official, Security, Trade
from pote.db.models import Official, Security, Trade
from pote.monitoring.disclosure_correlator import DisclosureCorrelator
logger = logging.getLogger(__name__)
@@ -60,9 +59,7 @@ class PatternDetector:
.all()
)
logger.info(
f"Analyzing {len(officials_with_trades)} officials with {min_trades}+ trades"
)
logger.info(f"Analyzing {len(officials_with_trades)} officials with {min_trades}+ trades")
rankings = []
@@ -70,9 +67,7 @@ class PatternDetector:
official_id, name, chamber, party, state, trade_count = official_data
# Get timing pattern
pattern = self.correlator.get_official_timing_pattern(
official_id, lookback_days
)
pattern = self.correlator.get_official_timing_pattern(official_id, lookback_days)
if pattern["trade_count"] == 0:
continue
@@ -128,9 +123,7 @@ class PatternDetector:
rankings = self.rank_officials_by_timing(lookback_days, min_trades=5)
# Filter for high suspicious rates
offenders = [
r for r in rankings if r["suspicious_rate"] >= min_suspicious_rate * 100
]
offenders = [r for r in rankings if r["suspicious_rate"] >= min_suspicious_rate * 100]
logger.info(
f"Found {len(offenders)} officials with {min_suspicious_rate*100}%+ suspicious trades"
@@ -155,9 +148,7 @@ class PatternDetector:
# Get tickers with enough trades
tickers_with_trades = (
self.session.query(
Security.ticker, func.count(Trade.id).label("trade_count")
)
self.session.query(Security.ticker, func.count(Trade.id).label("trade_count"))
.join(Trade)
.filter(Trade.transaction_date >= since_date)
.group_by(Security.ticker)
@@ -169,10 +160,8 @@ class PatternDetector:
ticker_patterns = []
for ticker, trade_count in tickers_with_trades:
analysis = self.correlator.get_ticker_timing_analysis(
ticker, lookback_days
)
for ticker, _trade_count in tickers_with_trades:
analysis = self.correlator.get_ticker_timing_analysis(ticker, lookback_days)
if analysis["trade_count"] == 0:
continue
@@ -199,9 +188,7 @@ class PatternDetector:
return ticker_patterns
def get_sector_timing_analysis(
self, lookback_days: int = 365
) -> dict[str, dict[str, Any]]:
def get_sector_timing_analysis(self, lookback_days: int = 365) -> dict[str, dict[str, Any]]:
"""
Analyze timing patterns by sector.
@@ -252,7 +239,7 @@ class PatternDetector:
sector_stats[sector]["suspicious_count"] += 1
# Calculate averages
for sector, stats in sector_stats.items():
for stats in sector_stats.values():
if stats["trade_count"] > 0:
stats["avg_timing_score"] = round(
stats["total_timing_score"] / stats["trade_count"], 2
@@ -266,9 +253,7 @@ class PatternDetector:
return sector_stats
def get_party_comparison(
self, lookback_days: int = 365
) -> dict[str, dict[str, Any]]:
def get_party_comparison(self, lookback_days: int = 365) -> dict[str, dict[str, Any]]:
"""
Compare timing patterns between political parties.
@@ -303,7 +288,7 @@ class PatternDetector:
party_stats[party]["officials"].append(ranking)
# Calculate averages
for party, stats in party_stats.items():
for stats in party_stats.values():
if stats["total_trades"] > 0:
stats["avg_timing_score"] = round(
stats["total_timing_score"] / stats["total_trades"], 2
@@ -356,5 +341,3 @@ class PatternDetector:
"sector_analysis": sector_analysis,
"party_comparison": party_comparison,
}
-2
View File
@@ -8,5 +8,3 @@ from .email_reporter import EmailReporter
from .report_generator import ReportGenerator
__all__ = ["EmailReporter", "ReportGenerator"]
+13 -16
View File
@@ -8,7 +8,6 @@ import logging
import smtplib
from email.mime.multipart import MIMEMultipart
from email.mime.text import MIMEText
from typing import List, Optional
from pote.config import settings
@@ -20,31 +19,30 @@ class EmailReporter:
def __init__(
self,
smtp_host: Optional[str] = None,
smtp_port: Optional[int] = None,
smtp_user: Optional[str] = None,
smtp_password: Optional[str] = None,
from_email: Optional[str] = None,
smtp_host: str | None = None,
smtp_port: int | None = None,
smtp_user: str | None = None,
smtp_password: str | None = None,
from_email: str | None = None,
):
"""
Initialize email reporter.
If parameters are not provided, will attempt to use settings from config.
"""
self.smtp_host = smtp_host or getattr(settings, "smtp_host", "localhost")
self.smtp_port = smtp_port or getattr(settings, "smtp_port", 587)
self.smtp_user = smtp_user or getattr(settings, "smtp_user", None)
self.smtp_password = smtp_password or getattr(settings, "smtp_password", None)
self.from_email = from_email or getattr(
settings, "from_email", "pote@localhost"
)
# Settings always defines these fields (with defaults), so direct access is safe.
self.smtp_host: str = smtp_host or settings.smtp_host
self.smtp_port: int = smtp_port or settings.smtp_port
self.smtp_user: str = smtp_user or settings.smtp_user
self.smtp_password: str = smtp_password or settings.smtp_password
self.from_email: str = from_email or settings.from_email or "pote@localhost"
def send_report(
self,
to_emails: List[str],
to_emails: list[str],
subject: str,
body_text: str,
body_html: Optional[str] = None,
body_html: str | None = None,
) -> bool:
"""
Send an email report.
@@ -113,4 +111,3 @@ class EmailReporter:
except Exception as e:
logger.error(f"SMTP connection test failed: {e}")
return False
+19 -27
View File
@@ -6,7 +6,7 @@ Generates formatted reports from database data.
import logging
from datetime import date, datetime, timedelta
from typing import Any, Dict, List, Optional
from typing import Any
from sqlalchemy import func
from sqlalchemy.orm import Session
@@ -27,8 +27,8 @@ class ReportGenerator:
self.detector = PatternDetector(session)
def generate_daily_summary(
self, report_date: Optional[date] = None, *, lookback_days: int = 1
) -> Dict[str, Any]:
self, report_date: date | None = None, *, lookback_days: int = 1
) -> dict[str, Any]:
"""
Generate a daily summary report.
@@ -69,7 +69,7 @@ class ReportGenerator:
)
# Get high-severity alerts
critical_alerts = [a for a in new_alerts if a.severity >= 7]
critical_alerts = [a for a in new_alerts if (a.severity or 0) >= 7]
# Get suspicious timing matches
suspicious_trades = []
@@ -110,7 +110,7 @@ class ReportGenerator:
"suspicious_trades": suspicious_trades,
}
def generate_weekly_summary(self) -> Dict[str, Any]:
def generate_weekly_summary(self) -> dict[str, Any]:
"""
Generate a weekly summary report.
@@ -121,9 +121,7 @@ class ReportGenerator:
# Most active officials
active_officials = (
self.session.query(
Official.name, func.count(Trade.id).label("trade_count")
)
self.session.query(Official.name, func.count(Trade.id).label("trade_count"))
.join(Trade)
.filter(Trade.filing_date >= week_ago)
.group_by(Official.id, Official.name)
@@ -134,9 +132,7 @@ class ReportGenerator:
# Most traded securities
active_securities = (
self.session.query(
Security.ticker, func.count(Trade.id).label("trade_count")
)
self.session.query(Security.ticker, func.count(Trade.id).label("trade_count"))
.join(Trade)
.filter(Trade.filing_date >= week_ago)
.group_by(Security.id, Security.ticker)
@@ -146,6 +142,8 @@ class ReportGenerator:
)
# Get top suspicious patterns
# Same fix as branch docs/deploy-email-closed: the old kwarg names never
# existed on identify_repeat_offenders and failed every weekly run.
repeat_offenders = self.detector.identify_repeat_offenders(
lookback_days=7, min_suspicious_rate=0.4
)
@@ -157,14 +155,13 @@ class ReportGenerator:
{"name": name, "trade_count": count} for name, count in active_officials
],
"most_traded_securities": [
{"ticker": ticker, "trade_count": count}
for ticker, count in active_securities
{"ticker": ticker, "trade_count": count} for ticker, count in active_securities
],
"repeat_offenders_count": len(repeat_offenders),
"repeat_offenders": repeat_offenders[:5], # Top 5
}
def format_as_text(self, report_data: Dict[str, Any], report_type: str) -> str:
def format_as_text(self, report_data: dict[str, Any], report_type: str) -> str:
"""
Format report data as plain text.
@@ -177,12 +174,11 @@ class ReportGenerator:
"""
if report_type == "daily":
return self._format_daily_text(report_data)
elif report_type == "weekly":
if report_type == "weekly":
return self._format_weekly_text(report_data)
else:
return str(report_data)
def _format_daily_text(self, data: Dict[str, Any]) -> str:
def _format_daily_text(self, data: dict[str, Any]) -> str:
"""Format daily report as plain text."""
if data.get("lookback_days", 1) > 1:
trades_label = (
@@ -245,7 +241,7 @@ class ReportGenerator:
return "\n".join(lines)
def _format_weekly_text(self, data: Dict[str, Any]) -> str:
def _format_weekly_text(self, data: dict[str, Any]) -> str:
"""Format weekly report as plain text."""
lines = [
"=" * 70,
@@ -264,9 +260,7 @@ class ReportGenerator:
lines.append(f"{security['ticker']}: {security['trade_count']} trades")
if data["repeat_offenders"]:
lines.extend(
["", f"⚠️ REPEAT OFFENDERS ({data['repeat_offenders_count']} total)"]
)
lines.extend(["", f"⚠️ REPEAT OFFENDERS ({data['repeat_offenders_count']} total)"])
for offender in data["repeat_offenders"]:
lines.append(
f"{offender['official_name']}: "
@@ -285,7 +279,7 @@ class ReportGenerator:
return "\n".join(lines)
def format_as_html(self, report_data: Dict[str, Any], report_type: str) -> str:
def format_as_html(self, report_data: dict[str, Any], report_type: str) -> str:
"""
Format report data as HTML.
@@ -298,12 +292,11 @@ class ReportGenerator:
"""
if report_type == "daily":
return self._format_daily_html(report_data)
elif report_type == "weekly":
if report_type == "weekly":
return self._format_weekly_html(report_data)
else:
return f"<pre>{report_data}</pre>"
def _format_daily_html(self, data: Dict[str, Any]) -> str:
def _format_daily_html(self, data: dict[str, Any]) -> str:
"""Format daily report as HTML."""
if data.get("lookback_days", 1) > 1:
new_trades_label = f"Trades Filed (last {data['lookback_days']} days):"
@@ -377,7 +370,7 @@ class ReportGenerator:
return html
def _format_weekly_html(self, data: Dict[str, Any]) -> str:
def _format_weekly_html(self, data: dict[str, Any]) -> str:
"""Format weekly report as HTML."""
html = f"""
<html>
@@ -443,4 +436,3 @@ class ReportGenerator:
"""
return html
+2 -2
View File
@@ -22,8 +22,8 @@ def test_db_session() -> Session:
engine = create_engine("sqlite:///:memory:", echo=False)
Base.metadata.create_all(engine)
TestSessionLocal = sessionmaker(bind=engine)
session = TestSessionLocal()
session_factory = sessionmaker(bind=engine)
session = session_factory()
yield session
+10 -6
View File
@@ -1,13 +1,14 @@
"""Tests for analytics module."""
import pytest
from datetime import date, timedelta
from decimal import Decimal
from pote.analytics.returns import ReturnCalculator
import pytest
from pote.analytics.benchmarks import BenchmarkComparison
from pote.analytics.metrics import PerformanceMetrics
from pote.db.models import Official, Security, Trade, Price
from pote.analytics.returns import ReturnCalculator
from pote.db.models import Price, Security, Trade
@pytest.fixture
@@ -80,7 +81,9 @@ def test_return_calculator_basic(test_db_session, sample_official, sample_securi
assert "exit_price" in result
def test_return_calculator_sell_trade(test_db_session, sample_official, sample_security, sample_prices):
def test_return_calculator_sell_trade(
test_db_session, sample_official, sample_security, sample_prices
):
session = test_db_session
"""Test return calculation for sell trade."""
trade = Trade(
@@ -154,7 +157,9 @@ def test_benchmark_comparison(test_db_session, sample_official, sample_security,
assert "beat_market" in result
def test_performance_metrics_official(test_db_session, sample_official, sample_security, sample_prices):
def test_performance_metrics_official(
test_db_session, sample_official, sample_security, sample_prices
):
session = test_db_session
"""Test official performance metrics."""
# Create multiple trades
@@ -265,4 +270,3 @@ def test_timing_analysis(test_db_session, sample_official, sample_security):
assert "avg_disclosure_lag_days" in timing
assert timing["avg_disclosure_lag_days"] > 0
+16 -12
View File
@@ -1,13 +1,14 @@
"""Integration tests for analytics with real-ish data."""
import pytest
from datetime import date, timedelta
from decimal import Decimal
from pote.analytics.returns import ReturnCalculator
import pytest
from pote.analytics.benchmarks import BenchmarkComparison
from pote.analytics.metrics import PerformanceMetrics
from pote.db.models import Official, Security, Trade, Price
from pote.analytics.returns import ReturnCalculator
from pote.db.models import Official, Price, Security, Trade
@pytest.fixture
@@ -129,7 +130,9 @@ def test_return_calculation_with_real_data(test_db_session, full_test_data):
# Entry around day 15, exit around day 105
# Expected return: (720 - 532.5) / 532.5 = ~35%
assert 30 < float(result["return_pct"]) < 50, f"Expected ~35% return, got {result['return_pct']}"
assert (
30 < float(result["return_pct"]) < 50
), f"Expected ~35% return, got {result['return_pct']}"
print(f"\n✅ NVDA 90-day return: {result['return_pct']:.2f}%")
print(f" Entry: ${result['entry_price']} on {result['transaction_date']}")
@@ -155,7 +158,7 @@ def test_benchmark_comparison_with_real_data(test_db_session, full_test_data):
assert result["beat_market"] is True
assert float(result["abnormal_return"]) > 10, "NVDA should have strong alpha vs SPY"
print(f"\n✅ Benchmark Comparison:")
print("\n✅ Benchmark Comparison:")
print(f" NVDA Return: {result['trade_return']:.2f}%")
print(f" SPY Return: {result['benchmark_return']:.2f}%")
print(f" Alpha: {result['abnormal_return']:+.2f}%")
@@ -201,7 +204,7 @@ def test_multiple_windows(test_db_session, full_test_data):
# Returns should generally increase with longer windows (given upward trend)
if 30 in results and 90 in results:
print(f"\n✅ Multiple Windows:")
print("\n✅ Multiple Windows:")
for window in [30, 60, 90]:
if window in results:
print(f" {window:3d} days: {results[window]['return_pct']:+7.2f}%")
@@ -217,7 +220,7 @@ def test_top_performers(test_db_session, full_test_data):
assert isinstance(top, list)
assert len(top) > 0
print(f"\n✅ Top Performers:")
print("\n✅ Top Performers:")
for i, perf in enumerate(top, 1):
if perf.get("trades_analyzed", 0) > 0:
print(f" {i}. {perf['name']:20s} | Alpha: {perf['avg_alpha']:+6.2f}%")
@@ -234,7 +237,7 @@ def test_system_statistics(test_db_session, full_test_data):
assert stats["total_trades"] >= 2
assert stats["total_securities"] >= 2
print(f"\n✅ System Statistics:")
print("\n✅ System Statistics:")
print(f" Officials: {stats['total_officials']}")
print(f" Trades: {stats['total_trades']}")
print(f" Securities: {stats['total_securities']}")
@@ -254,7 +257,7 @@ def test_disclosure_timing(test_db_session, full_test_data):
assert "avg_disclosure_lag_days" in timing
assert timing["avg_disclosure_lag_days"] > 0
print(f"\n✅ Disclosure Timing:")
print("\n✅ Disclosure Timing:")
print(f" Average Lag: {timing['avg_disclosure_lag_days']:.1f} days")
print(f" Median Lag: {timing['median_disclosure_lag_days']} days")
@@ -269,9 +272,11 @@ def test_sector_analysis(test_db_session, full_test_data):
assert isinstance(sectors, list)
if sectors:
print(f"\n✅ Sector Analysis:")
print("\n✅ Sector Analysis:")
for s in sectors:
print(f" {s['sector']:15s} | {s['trade_count']} trades | Alpha: {s['avg_alpha']:+6.2f}%")
print(
f" {s['sector']:15s} | {s['trade_count']} trades | Alpha: {s['avg_alpha']:+6.2f}%"
)
def test_edge_case_missing_exit_price(test_db_session, full_test_data):
@@ -328,4 +333,3 @@ def test_sell_trade_logic(test_db_session, full_test_data):
# Selling during uptrend = negative return
assert result["return_pct"] < 0, "Sell during uptrend should show negative return"
print(f"\n✅ Sell trade return correctly inverted: {result['return_pct']:.2f}%")
+16 -17
View File
@@ -1,11 +1,12 @@
"""Tests for disclosure correlation module."""
import pytest
from datetime import date, datetime, timedelta, timezone
from datetime import UTC, date, datetime
from decimal import Decimal
import pytest
from pote.db.models import MarketAlert, Official, Security, Trade
from pote.monitoring.disclosure_correlator import DisclosureCorrelator
from pote.db.models import Official, Security, Trade, MarketAlert
@pytest.fixture
@@ -38,7 +39,7 @@ def trade_with_alerts(test_db_session):
MarketAlert(
ticker="NVDA",
alert_type="unusual_volume",
timestamp=datetime(2024, 1, 10, 10, 30, tzinfo=timezone.utc), # 5 days before
timestamp=datetime(2024, 1, 10, 10, 30, tzinfo=UTC), # 5 days before
details={"multiplier": 3.5},
price=Decimal("490.00"),
volume=100000000,
@@ -48,7 +49,7 @@ def trade_with_alerts(test_db_session):
MarketAlert(
ticker="NVDA",
alert_type="price_spike",
timestamp=datetime(2024, 1, 12, 14, 15, tzinfo=timezone.utc), # 3 days before
timestamp=datetime(2024, 1, 12, 14, 15, tzinfo=UTC), # 3 days before
details={"change_pct": 5.5},
price=Decimal("505.00"),
volume=85000000,
@@ -58,7 +59,7 @@ def trade_with_alerts(test_db_session):
MarketAlert(
ticker="NVDA",
alert_type="high_volatility",
timestamp=datetime(2024, 1, 14, 16, 20, tzinfo=timezone.utc), # 1 day before
timestamp=datetime(2024, 1, 14, 16, 20, tzinfo=UTC), # 1 day before
details={"multiplier": 2.5},
price=Decimal("510.00"),
volume=90000000,
@@ -208,13 +209,13 @@ def test_calculate_timing_score_factors(test_db_session):
MarketAlert(
ticker="TEST",
alert_type="test",
timestamp=datetime(2024, 1, 10, 12, 0, tzinfo=timezone.utc),
timestamp=datetime(2024, 1, 10, 12, 0, tzinfo=UTC),
severity=3,
),
MarketAlert(
ticker="TEST",
alert_type="test",
timestamp=datetime(2024, 1, 11, 12, 0, tzinfo=timezone.utc),
timestamp=datetime(2024, 1, 11, 12, 0, tzinfo=UTC),
severity=4,
),
]
@@ -228,13 +229,13 @@ def test_calculate_timing_score_factors(test_db_session):
MarketAlert(
ticker="TEST",
alert_type="test",
timestamp=datetime(2024, 1, 13, 12, 0, tzinfo=timezone.utc), # Recent
timestamp=datetime(2024, 1, 13, 12, 0, tzinfo=UTC), # Recent
severity=9,
),
MarketAlert(
ticker="TEST",
alert_type="test",
timestamp=datetime(2024, 1, 14, 12, 0, tzinfo=timezone.utc), # Very recent
timestamp=datetime(2024, 1, 14, 12, 0, tzinfo=UTC), # Very recent
severity=8,
),
]
@@ -284,8 +285,7 @@ def test_analyze_recent_disclosures(test_db_session, trade_with_alerts, trade_wi
# Both trades were created "recently" (in fixture setup)
suspicious_trades = correlator.analyze_recent_disclosures(
days=365, # Wide window to catch test data
min_timing_score=50
days=365, min_timing_score=50 # Wide window to catch test data
)
# Should find at least the suspicious trade
@@ -384,7 +384,7 @@ def test_alerts_outside_lookback_window(test_db_session):
recent_alert = MarketAlert(
ticker="TEST",
alert_type="test",
timestamp=datetime(2024, 1, 13, 12, 0, tzinfo=timezone.utc),
timestamp=datetime(2024, 1, 13, 12, 0, tzinfo=UTC),
severity=7,
)
@@ -392,7 +392,7 @@ def test_alerts_outside_lookback_window(test_db_session):
old_alert = MarketAlert(
ticker="TEST",
alert_type="test",
timestamp=datetime(2023, 12, 6, 12, 0, tzinfo=timezone.utc),
timestamp=datetime(2023, 12, 6, 12, 0, tzinfo=UTC),
severity=8,
)
@@ -433,14 +433,14 @@ def test_different_ticker_alerts_excluded(test_db_session):
nvda_alert = MarketAlert(
ticker="NVDA",
alert_type="test",
timestamp=datetime(2024, 1, 10, 12, 0, tzinfo=timezone.utc),
timestamp=datetime(2024, 1, 10, 12, 0, tzinfo=UTC),
severity=7,
)
msft_alert = MarketAlert(
ticker="MSFT",
alert_type="test",
timestamp=datetime(2024, 1, 10, 12, 0, tzinfo=timezone.utc),
timestamp=datetime(2024, 1, 10, 12, 0, tzinfo=UTC),
severity=8,
)
@@ -452,4 +452,3 @@ def test_different_ticker_alerts_excluded(test_db_session):
assert len(alerts) == 1
assert alerts[0].ticker == "NVDA"
+3 -8
View File
@@ -5,6 +5,7 @@ Tests for database models.
from datetime import date
from decimal import Decimal
import pytest
from sqlalchemy import select
from pote.db.models import Price, Security, Trade
@@ -56,12 +57,9 @@ def test_unique_constraints(test_db_session, sample_security):
dup_security = Security(ticker="AAPL", name="Apple Duplicate")
test_db_session.add(dup_security)
try:
with pytest.raises(IntegrityError):
test_db_session.commit()
assert False, "Should have raised IntegrityError"
except IntegrityError:
test_db_session.rollback()
# Expected behavior
def test_price_unique_per_security_date(test_db_session, sample_security):
@@ -83,12 +81,9 @@ def test_price_unique_per_security_date(test_db_session, sample_security):
)
test_db_session.add(price2)
try:
with pytest.raises(IntegrityError):
test_db_session.commit()
assert False, "Should have raised IntegrityError"
except IntegrityError:
test_db_session.rollback()
# Expected behavior
def test_trade_queries(test_db_session, sample_official, sample_security):
+60 -30
View File
@@ -1,12 +1,13 @@
"""Tests for market monitoring module."""
import pytest
from datetime import date, datetime, timedelta, timezone
from datetime import UTC, date, datetime, timedelta
from decimal import Decimal
from pote.monitoring.market_monitor import MarketMonitor
import pytest
from pote.db.models import MarketAlert, Official, Security, Trade
from pote.monitoring.alert_manager import AlertManager
from pote.db.models import Official, Security, Trade, MarketAlert
from pote.monitoring.market_monitor import MarketMonitor
@pytest.fixture
@@ -31,21 +32,51 @@ def sample_congressional_trades(test_db_session):
# Create multiple trades (NVDA is most traded)
trades = [
Trade(official_id=pelosi.id, security_id=nvda.id, source="test",
transaction_date=date(2024, 1, 15), side="buy",
value_min=Decimal("15001"), value_max=Decimal("50000")),
Trade(official_id=pelosi.id, security_id=nvda.id, source="test",
transaction_date=date(2024, 2, 1), side="buy",
value_min=Decimal("15001"), value_max=Decimal("50000")),
Trade(official_id=tuberville.id, security_id=nvda.id, source="test",
transaction_date=date(2024, 2, 15), side="buy",
value_min=Decimal("50001"), value_max=Decimal("100000")),
Trade(official_id=pelosi.id, security_id=msft.id, source="test",
transaction_date=date(2024, 1, 20), side="sell",
value_min=Decimal("15001"), value_max=Decimal("50000")),
Trade(official_id=tuberville.id, security_id=aapl.id, source="test",
transaction_date=date(2024, 2, 10), side="buy",
value_min=Decimal("15001"), value_max=Decimal("50000")),
Trade(
official_id=pelosi.id,
security_id=nvda.id,
source="test",
transaction_date=date(2024, 1, 15),
side="buy",
value_min=Decimal("15001"),
value_max=Decimal("50000"),
),
Trade(
official_id=pelosi.id,
security_id=nvda.id,
source="test",
transaction_date=date(2024, 2, 1),
side="buy",
value_min=Decimal("15001"),
value_max=Decimal("50000"),
),
Trade(
official_id=tuberville.id,
security_id=nvda.id,
source="test",
transaction_date=date(2024, 2, 15),
side="buy",
value_min=Decimal("50001"),
value_max=Decimal("100000"),
),
Trade(
official_id=pelosi.id,
security_id=msft.id,
source="test",
transaction_date=date(2024, 1, 20),
side="sell",
value_min=Decimal("15001"),
value_max=Decimal("50000"),
),
Trade(
official_id=tuberville.id,
security_id=aapl.id,
source="test",
transaction_date=date(2024, 2, 10),
side="buy",
value_min=Decimal("15001"),
value_max=Decimal("50000"),
),
]
session.add_all(trades)
session.commit()
@@ -62,7 +93,7 @@ def sample_alerts(test_db_session):
"""Create sample market alerts."""
session = test_db_session
now = datetime.now(timezone.utc)
now = datetime.now(UTC)
alerts = [
MarketAlert(
@@ -145,7 +176,7 @@ def test_scan_watchlist_with_mock(test_db_session, sample_congressional_trades,
{
"ticker": ticker,
"alert_type": "unusual_volume",
"timestamp": datetime.now(timezone.utc),
"timestamp": datetime.now(UTC),
"details": {"multiplier": 3.5},
"price": Decimal("500.00"),
"volume": 100000000,
@@ -174,7 +205,7 @@ def test_save_alerts(test_db_session):
{
"ticker": "TSLA",
"alert_type": "price_spike",
"timestamp": datetime.now(timezone.utc),
"timestamp": datetime.now(UTC),
"details": {"change_pct": 7.5},
"price": Decimal("250.00"),
"volume": 75000000,
@@ -184,7 +215,7 @@ def test_save_alerts(test_db_session):
{
"ticker": "TSLA",
"alert_type": "unusual_volume",
"timestamp": datetime.now(timezone.utc),
"timestamp": datetime.now(UTC),
"details": {"multiplier": 4.0},
"price": Decimal("250.00"),
"volume": 120000000,
@@ -297,7 +328,7 @@ def test_alert_manager_filter_alerts(test_db_session, sample_alerts):
sample_alerts,
min_severity=4,
tickers=["NVDA"],
alert_types=["unusual_volume", "price_spike"]
alert_types=["unusual_volume", "price_spike"],
)
assert len(filtered) == 2
@@ -307,7 +338,7 @@ def test_alert_manager_generate_summary_text(test_db_session, sample_alerts):
session = test_db_session
alert_mgr = AlertManager(session)
report = alert_mgr.generate_summary_report(sample_alerts, format="text")
report = alert_mgr.generate_summary_report(sample_alerts, output_format="text")
assert "MARKET ACTIVITY ALERTS" in report
assert "3 Alerts" in report
@@ -321,7 +352,7 @@ def test_alert_manager_generate_summary_html(test_db_session, sample_alerts):
session = test_db_session
alert_mgr = AlertManager(session)
report = alert_mgr.generate_summary_report(sample_alerts, format="html")
report = alert_mgr.generate_summary_report(sample_alerts, output_format="html")
assert "<html>" in report
assert "<head>" in report
@@ -334,7 +365,7 @@ def test_alert_manager_empty_alerts(test_db_session):
session = test_db_session
alert_mgr = AlertManager(session)
report = alert_mgr.generate_summary_report([], format="text")
report = alert_mgr.generate_summary_report([], output_format="text")
assert "No alerts" in report
@@ -346,7 +377,7 @@ def test_market_alert_model(test_db_session):
alert = MarketAlert(
ticker="GOOGL",
alert_type="price_spike",
timestamp=datetime.now(timezone.utc),
timestamp=datetime.now(UTC),
details={"test": "data"},
price=Decimal("140.50"),
volume=25000000,
@@ -373,7 +404,7 @@ def test_alert_timestamp_filtering(test_db_session):
"""Test filtering alerts by timestamp."""
session = test_db_session
now = datetime.now(timezone.utc)
now = datetime.now(UTC)
# Create alerts at different times
old_alert = MarketAlert(
@@ -404,4 +435,3 @@ def test_alert_timestamp_filtering(test_db_session):
alerts_30_days = monitor.get_recent_alerts(days=30)
test_alerts = [a for a in alerts_30_days if a.ticker.startswith("TEST")]
assert len(test_alerts) == 2
+14 -21
View File
@@ -1,11 +1,12 @@
"""Tests for pattern detection module."""
import pytest
from datetime import date, datetime, timedelta, timezone
from datetime import UTC, date, datetime, timedelta
from decimal import Decimal
import pytest
from pote.db.models import MarketAlert, Official, Security, Trade
from pote.monitoring.pattern_detector import PatternDetector
from pote.db.models import Official, Security, Trade, MarketAlert
@pytest.fixture
@@ -31,7 +32,7 @@ def multiple_officials_with_patterns(test_db_session):
# Pelosi - Suspicious pattern (trades with alerts)
for i in range(5):
trade_date = date(2024, 1, 15) + timedelta(days=i*30)
trade_date = date(2024, 1, 15) + timedelta(days=i * 30)
# Create trade
trade = Trade(
@@ -52,16 +53,15 @@ def multiple_officials_with_patterns(test_db_session):
ticker="NVDA",
alert_type="unusual_volume",
timestamp=datetime.combine(
trade_date - timedelta(days=3+j),
datetime.min.time()
).replace(tzinfo=timezone.utc),
trade_date - timedelta(days=3 + j), datetime.min.time()
).replace(tzinfo=UTC),
severity=7 + j,
)
session.add(alert)
# Tuberville - Mixed pattern
for i in range(4):
trade_date = date(2024, 2, 1) + timedelta(days=i*30)
trade_date = date(2024, 2, 1) + timedelta(days=i * 30)
trade = Trade(
official_id=tuberville.id,
@@ -81,16 +81,15 @@ def multiple_officials_with_patterns(test_db_session):
ticker="MSFT",
alert_type="price_spike",
timestamp=datetime.combine(
trade_date - timedelta(days=5),
datetime.min.time()
).replace(tzinfo=timezone.utc),
trade_date - timedelta(days=5), datetime.min.time()
).replace(tzinfo=UTC),
severity=6,
)
session.add(alert)
# Clean trader - No suspicious activity
for i in range(3):
trade_date = date(2024, 3, 1) + timedelta(days=i*30)
trade_date = date(2024, 3, 1) + timedelta(days=i * 30)
trade = Trade(
official_id=clean_trader.id,
@@ -141,8 +140,7 @@ def test_identify_repeat_offenders(test_db_session, multiple_officials_with_patt
# Set low threshold to catch Pelosi (who has 100% suspicious rate)
offenders = detector.identify_repeat_offenders(
lookback_days=3650,
min_suspicious_rate=0.7 # 70%+
lookback_days=3650, min_suspicious_rate=0.7 # 70%+
)
# Should find at least Pelosi (all trades with alerts)
@@ -158,10 +156,7 @@ def test_analyze_ticker_patterns(test_db_session, multiple_officials_with_patter
session = test_db_session
detector = PatternDetector(session)
ticker_patterns = detector.analyze_ticker_patterns(
lookback_days=3650,
min_trades=3
)
ticker_patterns = detector.analyze_ticker_patterns(lookback_days=3650, min_trades=3)
assert isinstance(ticker_patterns, list)
assert len(ticker_patterns) >= 1 # At least NVDA should qualify
@@ -208,7 +203,7 @@ def test_get_party_comparison(test_db_session, multiple_officials_with_patterns)
assert len(party_stats) >= 2 # Democrat, Republican, Independent
# Check that we have data for each party
for party, stats in party_stats.items():
for stats in party_stats.values():
assert "official_count" in stats
assert "total_trades" in stats
assert "avg_timing_score" in stats
@@ -322,5 +317,3 @@ def test_party_stats_completeness(test_db_session, multiple_officials_with_patte
assert dem["official_count"] >= 1
assert dem["total_trades"] >= 5 # Pelosi has 5 trades
assert dem["total_suspicious"] > 0 # Pelosi has suspicious trades