Initial commit: POTE Phase 1 complete
- PR1: Project scaffold, DB models, price loader - PR2: Congressional trade ingestion (House Stock Watcher) - PR3: Security enrichment + deployment infrastructure - 37 passing tests, 87%+ coverage - Docker + Proxmox deployment ready - Complete documentation - Works 100% offline with fixtures
This commit is contained in:
ilia
@@ -0,0 +1,84 @@
|
||||
# MVP (Phase 1) — US Congress prototype
|
||||
|
||||
This document defines a **minimal viable research system** for ingesting U.S. Congress trade disclosures, storing them in a relational DB, joining to daily price data, and computing a small set of descriptive metrics.
|
||||
|
||||
## Non-goals (explicit)
|
||||
- No trading execution, brokerage integration, alerts for “buy/sell”, or portfolio automation.
|
||||
- No claims of insider information.
|
||||
- No promises of alpha; all outputs are descriptive analytics with caveats.
|
||||
|
||||
## MVP definition (what “done” means)
|
||||
The MVP is “done” when a researcher can:
|
||||
- Ingest recent U.S. Congress trade disclosures from at least **one** public source (e.g., QuiverQuant or FMP) into a DB.
|
||||
- Ingest daily prices for traded tickers (e.g., yfinance) into the DB.
|
||||
- Run a query/report that shows, for an official and date range:
|
||||
- trades (buy/sell, transaction + filing dates, amount/value range when available)
|
||||
- post-trade returns over fixed windows (e.g., 1M/3M/6M) and a simple benchmark (e.g., SPY) to produce **abnormal return**
|
||||
- Compute and store a small set of **risk/ethics flags** (rule-based, transparent, caveated).
|
||||
|
||||
## PR-sized rollout plan (sequence)
|
||||
|
||||
### PR 1 — Project scaffold + tooling (small, boring, reliable)
|
||||
- Create `src/` + `tests/` layout
|
||||
- Add `pyproject.toml` with formatting/lint/test tooling
|
||||
- Add `.env.example` + settings loader
|
||||
- Add `README` update: how to run tests, configure DB
|
||||
|
||||
### PR 2 — Database + schema (SQLAlchemy + Alembic)
|
||||
- SQLAlchemy models for:
|
||||
- `officials`
|
||||
- `securities`
|
||||
- `trades`
|
||||
- `prices`
|
||||
- `metrics_trade` (derived metrics per trade)
|
||||
- `metrics_official` (aggregates)
|
||||
- Alembic migration + SQLite dev default
|
||||
- Tests: model constraints + simple insert/query smoke tests
|
||||
|
||||
### PR 3 — API client: Congress trade disclosures (one source)
|
||||
- Implement a small client module (requests/httpx)
|
||||
- Add retry/backoff + basic rate limiting
|
||||
- Normalize raw payloads → internal dataclasses/pydantic models
|
||||
- Tests: unit tests with mocked HTTP responses
|
||||
|
||||
### PR 4 — ETL: upsert officials/securities/trades
|
||||
- Idempotent ETL job:
|
||||
- fetch recent disclosures
|
||||
- normalize
|
||||
- upsert into DB
|
||||
- Logging of counts (new/updated/skipped)
|
||||
- Tests: idempotency and upsert behavior with SQLite
|
||||
|
||||
### PR 5 — Price loader (daily bars)
|
||||
- Given tickers + date range: fetch prices (e.g., yfinance) and upsert
|
||||
- Basic caching:
|
||||
- don’t refetch days already present unless forced
|
||||
- fetch missing ranges only
|
||||
- Tests: caching behavior (mock provider)
|
||||
|
||||
### PR 6 — Metrics + first “research signals” (non-advice)
|
||||
- Compute per-trade:
|
||||
- forward returns (1M/3M/6M)
|
||||
- benchmark returns (SPY) and abnormal returns
|
||||
- Store to `metrics_trade`
|
||||
- Aggregate to `metrics_official`
|
||||
- Add **transparent flags** (examples):
|
||||
- `watch_large_trade`: above configurable value range threshold
|
||||
- `watch_fast_filing_gap`: long or suspicious filing gaps (descriptive)
|
||||
- `watch_sensitive_sector`: sector in a configurable list (research-only heuristic)
|
||||
- Tests: deterministic calculations on synthetic price series
|
||||
|
||||
### PR 7 — CLI / query helpers (research workflow)
|
||||
- CLI commands:
|
||||
- “show trades for official”
|
||||
- “top officials by average abnormal return (with sample size)”
|
||||
- “sector interest trend”
|
||||
- All outputs include: **“research only, not investment advice”**
|
||||
|
||||
## Key MVP decisions (defaults)
|
||||
- **DB**: SQLite by default for dev; Postgres supported via env.
|
||||
- **Time**: store all dates in ISO format; use timezone-aware datetimes where needed.
|
||||
- **Idempotency**: every ingestion and metric step can be re-run safely.
|
||||
- **Reproducibility**: record data source and raw identifiers for traceability.
|
||||
|
||||
|
||||
@@ -0,0 +1,57 @@
|
||||
# Architecture (target shape for Phase 1)
|
||||
|
||||
This is an intentionally simple architecture optimized for **clarity, idempotency, and testability**.
|
||||
|
||||
## High-level flow
|
||||
1. **Ingest disclosures** (public source API) → normalize → upsert to DB (`officials`, `securities`, `trades`)
|
||||
2. **Load market data** (daily prices) → upsert to DB (`prices`)
|
||||
3. **Compute metrics** (returns, benchmarks, aggregates) → write to DB (`metrics_trade`, `metrics_official`)
|
||||
4. **Query/report** via CLI (later: read-only API/dashboard)
|
||||
|
||||
## Proposed module layout (to be created)
|
||||
|
||||
```
|
||||
src/pote/
|
||||
__init__.py
|
||||
config.py # settings loader (.env), constants
|
||||
db/
|
||||
__init__.py
|
||||
session.py # engine + sessionmaker
|
||||
models.py # SQLAlchemy ORM models
|
||||
migrations/ # Alembic (added once models stabilize)
|
||||
clients/
|
||||
__init__.py
|
||||
quiver.py # QuiverQuant client (optional)
|
||||
fmp.py # Financial Modeling Prep client (optional)
|
||||
market_data.py # yfinance wrapper / other provider interface
|
||||
etl/
|
||||
__init__.py
|
||||
congress_trades.py # disclosure ingestion + upsert
|
||||
prices.py # price ingestion + upsert + caching
|
||||
analytics/
|
||||
__init__.py
|
||||
returns.py # return & abnormal return calculations
|
||||
signals.py # rule-based “flags” (transparent, caveated)
|
||||
aggregations.py # per-official summaries
|
||||
cli/
|
||||
__init__.py
|
||||
main.py # entrypoint for research queries
|
||||
tests/
|
||||
...
|
||||
```
|
||||
|
||||
## Design constraints (non-negotiable)
|
||||
- **Public data only**: every record must store `source` and enough IDs to trace back.
|
||||
- **No advice**: outputs and docs must avoid prescriptive language and include disclaimers.
|
||||
- **Idempotency**: ETL and metrics jobs must be safe to rerun.
|
||||
- **Separation of concerns**:
|
||||
- clients fetch raw data
|
||||
- etl normalizes + writes
|
||||
- analytics reads normalized data and writes derived tables
|
||||
|
||||
## Operational conventions
|
||||
- Logging: structured-ish logs with counts (fetched/inserted/updated/skipped).
|
||||
- Rate limits: conservative defaults; provide `--sleep`/`--max-requests` config as needed.
|
||||
- Config: one settings object with env var support; `.env.example` committed, `.env` ignored.
|
||||
|
||||
|
||||
@@ -0,0 +1,102 @@
|
||||
# Data model (normalized schema sketch)
|
||||
|
||||
This is the Phase 1 target schema. Exact fields may vary slightly by available source data; the goal is to keep raw ingestion **traceable** and analytics **reproducible**.
|
||||
|
||||
## Core tables
|
||||
|
||||
### `officials`
|
||||
Represents an individual official (starting with U.S. Congress).
|
||||
|
||||
Suggested fields:
|
||||
- `id` (PK)
|
||||
- `name` (string)
|
||||
- `chamber` (enum-like string: House/Senate/Unknown)
|
||||
- `party` (string, nullable)
|
||||
- `state` (string, nullable)
|
||||
- `identifiers` (JSON) — e.g., bioguide ID, source-specific IDs
|
||||
- `created_at`, `updated_at`
|
||||
|
||||
### `securities`
|
||||
Represents a traded instrument.
|
||||
|
||||
Suggested fields:
|
||||
- `id` (PK)
|
||||
- `ticker` (string, indexed, nullable) — some disclosures may be missing ticker
|
||||
- `name` (string, nullable)
|
||||
- `exchange` (string, nullable)
|
||||
- `sector` (string, nullable)
|
||||
- `identifiers` (JSON) — ISIN, CUSIP, etc (when available)
|
||||
- `created_at`, `updated_at`
|
||||
|
||||
### `trades`
|
||||
One disclosed transaction record.
|
||||
|
||||
Suggested fields:
|
||||
- `id` (PK)
|
||||
- `official_id` (FK → `officials.id`)
|
||||
- `security_id` (FK → `securities.id`)
|
||||
- `source` (string) — e.g., `quiver`, `fmp`, `house_disclosure`
|
||||
- `source_trade_id` (string, nullable) — unique if provided
|
||||
- `transaction_date` (date, nullable if unknown)
|
||||
- `filing_date` (date, nullable)
|
||||
- `side` (enum-like string: BUY/SELL/EXCHANGE/UNKNOWN)
|
||||
- `value_range_low` (numeric, nullable)
|
||||
- `value_range_high` (numeric, nullable)
|
||||
- `amount` (numeric, nullable) — shares/contracts if available
|
||||
- `currency` (string, default USD)
|
||||
- `quality_flags` (JSON) — parse warnings, missing fields, etc
|
||||
- `raw` (JSON) — optional: raw payload snapshot for traceability
|
||||
- `created_at`, `updated_at`
|
||||
|
||||
Uniqueness strategy (typical):
|
||||
- unique constraint on (`source`, `source_trade_id`) when `source_trade_id` exists
|
||||
- otherwise a best-effort dedupe key (official, security, transaction_date, side, value_range_high, filing_date)
|
||||
|
||||
### `prices`
|
||||
Daily OHLCV for a ticker.
|
||||
|
||||
Suggested fields:
|
||||
- `id` (PK) or composite key
|
||||
- `ticker` (string, indexed)
|
||||
- `date` (date, indexed)
|
||||
- `open`, `high`, `low`, `close` (numeric)
|
||||
- `adj_close` (numeric, nullable)
|
||||
- `volume` (bigint, nullable)
|
||||
- `source` (string) — e.g., `yfinance`
|
||||
- `created_at`, `updated_at`
|
||||
|
||||
Unique constraint:
|
||||
- (`ticker`, `date`, `source`)
|
||||
|
||||
## Derived tables
|
||||
|
||||
### `metrics_trade`
|
||||
Per-trade derived analytics (computed after prices are loaded).
|
||||
|
||||
Suggested fields:
|
||||
- `id` (PK)
|
||||
- `trade_id` (FK → `trades.id`, unique)
|
||||
- forward returns: `ret_1m`, `ret_3m`, `ret_6m`
|
||||
- benchmark returns: `bm_ret_1m`, `bm_ret_3m`, `bm_ret_6m`
|
||||
- abnormal returns: `abret_1m`, `abret_3m`, `abret_6m`
|
||||
- `calc_version` (string) — allows recomputation while tracking methodology
|
||||
- `created_at`, `updated_at`
|
||||
|
||||
### `metrics_official`
|
||||
Aggregate metrics per official.
|
||||
|
||||
Suggested fields:
|
||||
- `id` (PK)
|
||||
- `official_id` (FK → `officials.id`, unique)
|
||||
- `n_trades`, `n_buys`, `n_sells`
|
||||
- average/median abnormal returns for buys (by window) + sample sizes
|
||||
- `cluster_label` (nullable)
|
||||
- `flags` (JSON) — descriptive risk/ethics flags + supporting metrics
|
||||
- `calc_version`
|
||||
- `created_at`, `updated_at`
|
||||
|
||||
## Notes on time and lags
|
||||
- Disclosures often have a filing delay; keep **both** `transaction_date` and `filing_date`.
|
||||
- When doing “event windows”, prefer windows relative to `transaction_date`, but also compute/record **disclosure lag** as a descriptive attribute.
|
||||
|
||||
|
||||
@@ -0,0 +1,53 @@
|
||||
# Data sources (public) + limitations
|
||||
|
||||
POTE only uses **lawfully available public data**. This project is for **private research** and produces **descriptive analytics** (not investment advice).
|
||||
|
||||
## Candidate sources (Phase 1)
|
||||
|
||||
### U.S. Congress trading disclosures
|
||||
- **QuiverQuant (API)**: provides congressional trading data (availability depends on plan/keys).
|
||||
- **Financial Modeling Prep (FMP)**: provides endpoints related to congressional trading and other market metadata (availability depends on plan/keys).
|
||||
- **Official disclosure sources** (future): House/Senate disclosure filings where accessible and lawful to process.
|
||||
|
||||
POTE will treat source data as “best effort” and store:
|
||||
- `source` (where it came from)
|
||||
- `source_trade_id` (if provided)
|
||||
- `raw` payload snapshot (optional, for traceability)
|
||||
- `quality_flags` describing parse/coverage issues
|
||||
|
||||
### Daily price data
|
||||
- **yfinance** (Yahoo finance wrapper) for daily OHLCV (research use; subject to availability and terms).
|
||||
- Alternative provider adapters can be added later (e.g., Stooq, AlphaVantage, Polygon, etc. as configured by the user).
|
||||
|
||||
## Known limitations / pitfalls
|
||||
|
||||
### Disclosure quality and ambiguity
|
||||
- **Tickers may be missing or wrong**; some disclosures list company names only or broad funds.
|
||||
- Transactions may be **value ranges** rather than exact amounts.
|
||||
- Some entries may reflect **family accounts** or managed accounts depending on disclosure details.
|
||||
- Duplicate records can occur across sources; deduplication is probabilistic when no unique ID exists.
|
||||
|
||||
### Timing and “lag”
|
||||
- Trades are often disclosed **after** the transaction date. Any analysis must account for:
|
||||
- transaction date
|
||||
- filing date
|
||||
- **disclosure lag** (filing - transaction)
|
||||
|
||||
### Survivorship / coverage
|
||||
- Some data providers may have incomplete histories or change coverage over time.
|
||||
- Price history may be missing for delisted tickers or corporate actions.
|
||||
|
||||
### Interpretation risks
|
||||
- Correlation is not causation; return outcomes do not imply intent or information access.
|
||||
- High abnormal returns can occur by chance; small samples are especially noisy.
|
||||
|
||||
## Source governance in this repo
|
||||
- No scraping that violates terms or access controls.
|
||||
- No bypassing paywalls, authentication, or restrictions.
|
||||
- When adding a new source, document:
|
||||
- endpoint/coverage
|
||||
- required API keys / limits
|
||||
- normalization mapping to the internal schema
|
||||
- known quirks
|
||||
|
||||
|
||||
@@ -0,0 +1 @@
|
||||
|
||||
@@ -0,0 +1,40 @@
|
||||
# Dev setup (conventions; code scaffolding comes next)
|
||||
|
||||
This doc sets the conventions we’ll implement in the first “code PRs”.
|
||||
|
||||
## Python + layout
|
||||
- Use Python 3.x
|
||||
- Source layout: `src/` + `tests/`
|
||||
- Prefer type hints and docstrings
|
||||
|
||||
## Configuration
|
||||
- Store secrets in `.env` (not committed).
|
||||
- Commit a `.env.example` documenting required variables.
|
||||
|
||||
Expected variables (initial):
|
||||
- `POTE_DB_URL` (e.g., `sqlite:///./pote.db` or Postgres URL)
|
||||
- `QUIVER_API_KEY` (optional, if using QuiverQuant)
|
||||
- `FMP_API_KEY` (optional, if using Financial Modeling Prep)
|
||||
|
||||
## Database
|
||||
- Default dev: SQLite for fast local iteration.
|
||||
- Support Postgres for “real” runs and larger datasets.
|
||||
- Migrations: Alembic (once models are in place).
|
||||
|
||||
## Testing
|
||||
- `pytest` for unit/integration tests
|
||||
- Prefer:
|
||||
- HTTP clients tested with mocked responses
|
||||
- DB tests using SQLite in a temp file or in-memory where possible
|
||||
|
||||
## Logging
|
||||
- Use standard `logging` with consistent, parseable messages.
|
||||
- ETL jobs should log counts: fetched/inserted/updated/skipped.
|
||||
|
||||
## PR sizing guideline
|
||||
Each PR should:
|
||||
- implement one coherent piece (schema, one client, one ETL, one metric module)
|
||||
- include tests
|
||||
- include minimal docs updates (if it changes behavior)
|
||||
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
# Free Testing: Data Sources & Sample Data Strategies
|
||||
|
||||
## Your Question: "How can we test for free?"
|
||||
|
||||
Great question! Here are multiple strategies for testing the full pipeline **without paid API keys**:
|
||||
|
||||
---
|
||||
|
||||
## Strategy 1: Mock/Fixture Data (Current Approach ✅)
|
||||
|
||||
**What we already have:**
|
||||
- `tests/conftest.py` creates in-memory SQLite DB with sample officials, securities, trades
|
||||
- Unit tests use mocked `yfinance` responses (see `test_price_loader.py`)
|
||||
- **Cost**: $0
|
||||
- **Coverage**: Models, DB logic, ETL transforms, analytics calculations
|
||||
|
||||
**Pros**: Fast, deterministic, no network, tests edge cases
|
||||
**Cons**: Doesn't validate real API behavior or data quality
|
||||
|
||||
---
|
||||
|
||||
## Strategy 2: Free Public Congressional Trade Data
|
||||
|
||||
### Option A: **House Stock Watcher** (Community Project)
|
||||
- **URL**: https://housestockwatcher.com/
|
||||
- **Format**: Web scraping (no official API, but RSS feed available)
|
||||
- **Data**: Real-time congressional trades (House & Senate)
|
||||
- **License**: Public domain (scraped from official disclosures)
|
||||
- **Cost**: $0
|
||||
- **How to use**:
|
||||
1. Scrape the RSS feed or JSON data from their GitHub repo
|
||||
2. Parse into our `trades` schema
|
||||
3. Use as integration test fixture
|
||||
|
||||
**Example**:
|
||||
```python
|
||||
# They have a JSON API endpoint (unofficial but free)
|
||||
import httpx
|
||||
resp = httpx.get("https://housestockwatcher.com/api/all_transactions")
|
||||
trades = resp.json()
|
||||
```
|
||||
|
||||
### Option B: **Senate Stock Watcher** API
|
||||
- **URL**: https://senatestockwatcher.com/
|
||||
- Similar to House Stock Watcher, community-maintained
|
||||
- Free JSON endpoints
|
||||
|
||||
### Option C: **Official Senate eFD** (Electronic Financial Disclosures)
|
||||
- **URL**: https://efdsearch.senate.gov/search/
|
||||
- **Format**: Web forms (no API, requires scraping)
|
||||
- **Cost**: $0, but requires building a scraper
|
||||
- **Data**: Official Senate disclosures (PTRs)
|
||||
|
||||
### Option D: **Quiver Quantitative Free Tier**
|
||||
- **URL**: https://www.quiverquant.com/
|
||||
- **Free tier**: 500 API calls/month (limited but usable for testing)
|
||||
- **Signup**: Email + API key (free)
|
||||
- **Data**: Congress, Senate, House trades + insider trades
|
||||
- **Docs**: https://api.quiverquant.com/docs
|
||||
|
||||
**Integration test example**:
|
||||
```python
|
||||
# Set QUIVERQUANT_API_KEY in .env for integration tests
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.skipif(not os.getenv("QUIVERQUANT_API_KEY"), reason="No API key")
|
||||
def test_quiver_live_fetch():
|
||||
client = QuiverClient(api_key=os.getenv("QUIVERQUANT_API_KEY"))
|
||||
trades = client.fetch_recent_trades(limit=10)
|
||||
assert len(trades) > 0
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Strategy 3: Use Sample/Historical Datasets
|
||||
|
||||
### Option A: **Pre-downloaded CSV Snapshots**
|
||||
1. Manually download 1-2 weeks of data from House/Senate Stock Watcher
|
||||
2. Store in `tests/fixtures/sample_trades.csv`
|
||||
3. Load in integration tests
|
||||
|
||||
**Example**:
|
||||
```python
|
||||
import pandas as pd
|
||||
from pathlib import Path
|
||||
|
||||
def test_etl_with_real_data():
|
||||
csv_path = Path(__file__).parent / "fixtures" / "sample_trades.csv"
|
||||
df = pd.read_csv(csv_path)
|
||||
# Run ETL pipeline
|
||||
loader = TradeLoader(session)
|
||||
loader.ingest_trades(df)
|
||||
# Assert trades were stored correctly
|
||||
```
|
||||
|
||||
### Option B: **Kaggle Datasets**
|
||||
- Search for "congressional stock trades" on Kaggle
|
||||
- Example: https://www.kaggle.com/datasets (check for recent uploads)
|
||||
- Download CSV, store in `tests/fixtures/`
|
||||
|
||||
---
|
||||
|
||||
## Strategy 4: Hybrid Testing (Recommended 🌟)
|
||||
|
||||
**Combine all strategies**:
|
||||
|
||||
1. **Unit tests** (fast, always run):
|
||||
- Use mocked data for models, ETL, analytics
|
||||
- `pytest tests/` (current setup)
|
||||
|
||||
2. **Integration tests** (optional, gated by env var):
|
||||
```python
|
||||
@pytest.mark.integration
|
||||
@pytest.mark.skipif(not os.getenv("ENABLE_LIVE_TESTS"), reason="Skipping live tests")
|
||||
def test_live_quiver_api():
|
||||
# Hits real Quiver API (free tier)
|
||||
pass
|
||||
```
|
||||
|
||||
3. **Fixture-based tests** (real data shape, no network):
|
||||
- Store 100 real trades in `tests/fixtures/sample_trades.json`
|
||||
- Test ETL, analytics, edge cases
|
||||
|
||||
4. **Manual smoke tests** (dev only):
|
||||
- `python scripts/fetch_sample_prices.py` (uses yfinance, free)
|
||||
- `python scripts/ingest_house_watcher.py` (once we build it)
|
||||
|
||||
---
|
||||
|
||||
## Recommended Next Steps
|
||||
|
||||
### For PR2 (Congress Trade Ingestion):
|
||||
1. **Build a House Stock Watcher scraper** (free, no API key needed)
|
||||
- Module: `src/pote/ingestion/house_watcher.py`
|
||||
- Scrape their RSS or JSON endpoint
|
||||
- Parse into `Trade` model
|
||||
- Store 100 sample trades in `tests/fixtures/`
|
||||
|
||||
2. **Add integration test marker**:
|
||||
```toml
|
||||
# pyproject.toml
|
||||
[tool.pytest.ini_options]
|
||||
markers = [
|
||||
"integration: marks tests as integration tests (require DB/network)",
|
||||
"slow: marks tests as slow",
|
||||
"live: requires external API/network (use --live flag)",
|
||||
]
|
||||
```
|
||||
|
||||
3. **Make PR2 testable without paid APIs**:
|
||||
```bash
|
||||
# Unit tests (always pass, use mocks)
|
||||
pytest tests/ -m "not integration"
|
||||
|
||||
# Integration tests (optional, use fixtures or free APIs)
|
||||
pytest tests/ -m integration
|
||||
|
||||
# Live tests (only if you have API keys)
|
||||
QUIVERQUANT_API_KEY=xxx pytest tests/ -m live
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cost Comparison
|
||||
|
||||
| Source | Free Tier | Paid Tier | Best For |
|
||||
|--------|-----------|-----------|----------|
|
||||
| **yfinance** | Unlimited | N/A | Prices (already working ✅) |
|
||||
| **House Stock Watcher** | Unlimited scraping | N/A | Free trades (best option) |
|
||||
| **Quiver Free** | 500 calls/mo | $30/mo (5k calls) | Testing, not production |
|
||||
| **FMP Free** | 250 calls/day | $15/mo | Alternative for trades |
|
||||
| **Mock data** | ∞ | N/A | Unit tests |
|
||||
|
||||
---
|
||||
|
||||
## Bottom Line
|
||||
|
||||
**You can build and test the entire system for $0** by:
|
||||
1. Using **House/Senate Stock Watcher** for real trade data (free, unlimited)
|
||||
2. Using **yfinance** for prices (already working)
|
||||
3. Storing **fixture snapshots** for regression tests
|
||||
4. Optionally using **Quiver free tier** (500 calls/mo) for validation
|
||||
|
||||
**No paid API required until you want:**
|
||||
- Production-grade rate limits
|
||||
- Historical data beyond 1-2 years
|
||||
- Official support/SLAs
|
||||
|
||||
---
|
||||
|
||||
## Example: Building a Free Trade Scraper (PR2)
|
||||
|
||||
```python
|
||||
# src/pote/ingestion/house_watcher.py
|
||||
import httpx
|
||||
from datetime import date
|
||||
|
||||
class HouseWatcherClient:
|
||||
"""Free congressional trade scraper."""
|
||||
|
||||
BASE_URL = "https://housestockwatcher.com"
|
||||
|
||||
def fetch_recent_trades(self, days: int = 7) -> list[dict]:
|
||||
"""Scrape recent trades (free, no API key)."""
|
||||
resp = httpx.get(f"{self.BASE_URL}/api/all_transactions")
|
||||
resp.raise_for_status()
|
||||
|
||||
trades = resp.json()
|
||||
# Filter to last N days, normalize to our schema
|
||||
return [self._normalize(t) for t in trades[:100]]
|
||||
|
||||
def _normalize(self, raw: dict) -> dict:
|
||||
"""Convert HouseWatcher format to our Trade schema."""
|
||||
return {
|
||||
"official_name": raw["representative"],
|
||||
"ticker": raw["ticker"],
|
||||
"transaction_date": raw["transaction_date"],
|
||||
"filing_date": raw["disclosure_date"],
|
||||
"side": "buy" if "Purchase" in raw["type"] else "sell",
|
||||
"value_min": raw.get("amount_min"),
|
||||
"value_max": raw.get("amount_max"),
|
||||
"source": "house_watcher",
|
||||
}
|
||||
```
|
||||
|
||||
Let me know if you want me to implement this scraper now for PR2! 🚀
|
||||
|
||||
@@ -0,0 +1,448 @@
|
||||
# Deployment Guide
|
||||
|
||||
## Deployment Options
|
||||
|
||||
POTE can be deployed in several ways depending on your needs:
|
||||
|
||||
1. **Local Development** (SQLite) - What you have now ✅
|
||||
2. **Single Server** (PostgreSQL + cron jobs)
|
||||
3. **Docker** (Containerized, easy to move)
|
||||
4. **Cloud** (AWS/GCP/Azure with managed DB)
|
||||
|
||||
---
|
||||
|
||||
## Option 1: Local Development (Current Setup) ✅
|
||||
|
||||
**You're already running this!**
|
||||
|
||||
```bash
|
||||
# Setup (done)
|
||||
make install
|
||||
source venv/bin/activate
|
||||
make migrate
|
||||
|
||||
# Ingest data
|
||||
python scripts/ingest_from_fixtures.py # Offline
|
||||
python scripts/fetch_congressional_trades.py --days 30 # With internet
|
||||
|
||||
# Query
|
||||
python
|
||||
>>> from pote.db import SessionLocal
|
||||
>>> from pote.db.models import Official
|
||||
>>> with SessionLocal() as session:
|
||||
... officials = session.query(Official).all()
|
||||
... print(f"Total officials: {len(officials)}")
|
||||
```
|
||||
|
||||
**Pros**: Simple, fast, no costs
|
||||
**Cons**: Local only, SQLite limitations for heavy queries
|
||||
|
||||
---
|
||||
|
||||
## Option 2: Single Server with PostgreSQL
|
||||
|
||||
### Setup PostgreSQL
|
||||
|
||||
```bash
|
||||
# Install PostgreSQL (Ubuntu/Debian)
|
||||
sudo apt update
|
||||
sudo apt install postgresql postgresql-contrib
|
||||
|
||||
# Create database
|
||||
sudo -u postgres psql
|
||||
postgres=# CREATE DATABASE pote;
|
||||
postgres=# CREATE USER poteuser WITH PASSWORD 'your_secure_password';
|
||||
postgres=# GRANT ALL PRIVILEGES ON DATABASE pote TO poteuser;
|
||||
postgres=# \q
|
||||
```
|
||||
|
||||
### Update Configuration
|
||||
|
||||
```bash
|
||||
# Edit .env
|
||||
DATABASE_URL=postgresql://poteuser:your_secure_password@localhost:5432/pote
|
||||
|
||||
# Run migrations
|
||||
source venv/bin/activate
|
||||
make migrate
|
||||
```
|
||||
|
||||
### Schedule Regular Ingestion
|
||||
|
||||
```bash
|
||||
# Add to crontab: crontab -e
|
||||
|
||||
# Fetch trades daily at 6 AM
|
||||
0 6 * * * cd /path/to/pote && /path/to/pote/venv/bin/python scripts/fetch_congressional_trades.py --days 7 >> /var/log/pote/trades.log 2>&1
|
||||
|
||||
# Enrich securities weekly on Sunday at 3 AM
|
||||
0 3 * * 0 cd /path/to/pote && /path/to/pote/venv/bin/python scripts/enrich_securities.py >> /var/log/pote/enrich.log 2>&1
|
||||
|
||||
# Fetch prices for all tickers daily at 7 AM
|
||||
0 7 * * * cd /path/to/pote && /path/to/pote/venv/bin/python scripts/update_all_prices.py >> /var/log/pote/prices.log 2>&1
|
||||
```
|
||||
|
||||
**Pros**: Production-ready, full SQL features, scheduled jobs
|
||||
**Cons**: Requires server management, PostgreSQL setup
|
||||
|
||||
---
|
||||
|
||||
## Option 3: Docker Deployment
|
||||
|
||||
### Create Dockerfile
|
||||
|
||||
```dockerfile
|
||||
# Dockerfile
|
||||
FROM python:3.11-slim
|
||||
|
||||
WORKDIR /app
|
||||
|
||||
# Install system dependencies
|
||||
RUN apt-get update && apt-get install -y \
|
||||
gcc \
|
||||
postgresql-client \
|
||||
&& rm -rf /var/lib/apt/lists/*
|
||||
|
||||
# Copy project files
|
||||
COPY pyproject.toml .
|
||||
COPY src/ src/
|
||||
COPY alembic/ alembic/
|
||||
COPY alembic.ini .
|
||||
COPY scripts/ scripts/
|
||||
|
||||
# Install Python dependencies
|
||||
RUN pip install --no-cache-dir -e .
|
||||
|
||||
# Run migrations on startup
|
||||
CMD ["sh", "-c", "alembic upgrade head && python scripts/fetch_congressional_trades.py --days 30"]
|
||||
```
|
||||
|
||||
### Docker Compose Setup
|
||||
|
||||
```yaml
|
||||
# docker-compose.yml
|
||||
version: '3.8'
|
||||
|
||||
services:
|
||||
db:
|
||||
image: postgres:15
|
||||
environment:
|
||||
POSTGRES_DB: pote
|
||||
POSTGRES_USER: poteuser
|
||||
POSTGRES_PASSWORD: ${POSTGRES_PASSWORD}
|
||||
volumes:
|
||||
- postgres_data:/var/lib/postgresql/data
|
||||
ports:
|
||||
- "5432:5432"
|
||||
|
||||
pote:
|
||||
build: .
|
||||
environment:
|
||||
DATABASE_URL: postgresql://poteuser:${POSTGRES_PASSWORD}@db:5432/pote
|
||||
QUIVERQUANT_API_KEY: ${QUIVERQUANT_API_KEY}
|
||||
FMP_API_KEY: ${FMP_API_KEY}
|
||||
depends_on:
|
||||
- db
|
||||
volumes:
|
||||
- ./logs:/app/logs
|
||||
|
||||
# Optional: FastAPI backend (Phase 3)
|
||||
api:
|
||||
build: .
|
||||
command: uvicorn pote.api.main:app --host 0.0.0.0 --port 8000
|
||||
environment:
|
||||
DATABASE_URL: postgresql://poteuser:${POSTGRES_PASSWORD}@db:5432/pote
|
||||
depends_on:
|
||||
- db
|
||||
ports:
|
||||
- "8000:8000"
|
||||
|
||||
volumes:
|
||||
postgres_data:
|
||||
```
|
||||
|
||||
### Deploy with Docker
|
||||
|
||||
```bash
|
||||
# Create .env file
|
||||
cat > .env << EOF
|
||||
POSTGRES_PASSWORD=your_secure_password
|
||||
DATABASE_URL=postgresql://poteuser:your_secure_password@db:5432/pote
|
||||
QUIVERQUANT_API_KEY=
|
||||
FMP_API_KEY=
|
||||
EOF
|
||||
|
||||
# Build and run
|
||||
docker-compose up -d
|
||||
|
||||
# Run migrations
|
||||
docker-compose exec pote alembic upgrade head
|
||||
|
||||
# Ingest data
|
||||
docker-compose exec pote python scripts/fetch_congressional_trades.py --days 30
|
||||
|
||||
# View logs
|
||||
docker-compose logs -f pote
|
||||
```
|
||||
|
||||
**Pros**: Portable, isolated, easy to deploy anywhere
|
||||
**Cons**: Requires Docker knowledge, slightly more complex
|
||||
|
||||
---
|
||||
|
||||
## Option 4: Cloud Deployment (AWS Example)
|
||||
|
||||
### AWS Architecture
|
||||
|
||||
```
|
||||
┌─────────────────┐
|
||||
│ EC2 Instance │
|
||||
│ - Python app │
|
||||
│ - Cron jobs │
|
||||
└────────┬────────┘
|
||||
│
|
||||
▼
|
||||
┌─────────────────┐
|
||||
│ RDS (Postgres)│
|
||||
│ - Managed DB │
|
||||
└─────────────────┘
|
||||
```
|
||||
|
||||
### Setup Steps
|
||||
|
||||
1. **Create RDS PostgreSQL Instance**
|
||||
- Go to AWS RDS Console
|
||||
- Create PostgreSQL 15 database
|
||||
- Note endpoint: `pote-db.xxxxx.us-east-1.rds.amazonaws.com`
|
||||
- Security group: Allow port 5432 from EC2
|
||||
|
||||
2. **Launch EC2 Instance**
|
||||
```bash
|
||||
# SSH into EC2
|
||||
ssh -i your-key.pem ubuntu@your-ec2-ip
|
||||
|
||||
# Install dependencies
|
||||
sudo apt update
|
||||
sudo apt install python3.11 python3-pip git
|
||||
|
||||
# Clone repo
|
||||
git clone <your-repo>
|
||||
cd pote
|
||||
|
||||
# Setup
|
||||
python3 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install -e .
|
||||
|
||||
# Configure
|
||||
cat > .env << EOF
|
||||
DATABASE_URL=postgresql://poteuser:password@pote-db.xxxxx.us-east-1.rds.amazonaws.com:5432/pote
|
||||
EOF
|
||||
|
||||
# Run migrations
|
||||
alembic upgrade head
|
||||
|
||||
# Setup cron jobs
|
||||
crontab -e
|
||||
# (Add the cron jobs from Option 2)
|
||||
```
|
||||
|
||||
3. **Optional: Use AWS Lambda for scheduled jobs**
|
||||
- Package app as Lambda function
|
||||
- Use EventBridge to trigger daily
|
||||
- Cheaper for infrequent jobs
|
||||
|
||||
**Pros**: Scalable, managed database, reliable
|
||||
**Cons**: Costs money (~$20-50/mo for small RDS + EC2)
|
||||
|
||||
---
|
||||
|
||||
## Option 5: Fly.io / Railway / Render (Easiest Cloud)
|
||||
|
||||
### Fly.io Example
|
||||
|
||||
```bash
|
||||
# Install flyctl
|
||||
curl -L https://fly.io/install.sh | sh
|
||||
|
||||
# Login
|
||||
flyctl auth login
|
||||
|
||||
# Create fly.toml
|
||||
cat > fly.toml << EOF
|
||||
app = "pote-research"
|
||||
|
||||
[build]
|
||||
builder = "paketobuildpacks/builder:base"
|
||||
|
||||
[env]
|
||||
PORT = "8080"
|
||||
|
||||
[[services]]
|
||||
internal_port = 8080
|
||||
protocol = "tcp"
|
||||
|
||||
[[services.ports]]
|
||||
port = 80
|
||||
|
||||
[postgres]
|
||||
app = "pote-db"
|
||||
EOF
|
||||
|
||||
# Create Postgres
|
||||
flyctl postgres create --name pote-db
|
||||
|
||||
# Deploy
|
||||
flyctl deploy
|
||||
|
||||
# Set secrets
|
||||
flyctl secrets set DATABASE_URL="postgres://..."
|
||||
```
|
||||
|
||||
**Pros**: Simple, cheap ($5-10/mo), automated deployments
|
||||
**Cons**: Limited control, may need to adapt code
|
||||
|
||||
---
|
||||
|
||||
## Production Checklist
|
||||
|
||||
Before deploying to production:
|
||||
|
||||
### Security
|
||||
- [ ] Change all default passwords
|
||||
- [ ] Use environment variables for secrets (never commit `.env`)
|
||||
- [ ] Enable SSL for database connections
|
||||
- [ ] Set up firewall rules (only allow necessary ports)
|
||||
- [ ] Use HTTPS if exposing API/dashboard
|
||||
|
||||
### Reliability
|
||||
- [ ] Set up database backups (daily)
|
||||
- [ ] Configure logging (centralized if possible)
|
||||
- [ ] Monitor disk space (especially for SQLite)
|
||||
- [ ] Set up error alerts (email/Slack on failures)
|
||||
- [ ] Test recovery from backup
|
||||
|
||||
### Performance
|
||||
- [ ] Index frequently queried columns (already done in models)
|
||||
- [ ] Use connection pooling for PostgreSQL
|
||||
- [ ] Cache frequently accessed data
|
||||
- [ ] Limit API rate if exposing publicly
|
||||
|
||||
### Compliance
|
||||
- [ ] Review data retention policy
|
||||
- [ ] Add disclaimers to any UI ("research only, not advice")
|
||||
- [ ] Document data sources and update frequency
|
||||
- [ ] Keep audit logs of data ingestion
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Logs
|
||||
|
||||
### Basic Logging Setup
|
||||
|
||||
```python
|
||||
# Add to scripts/fetch_congressional_trades.py
|
||||
import logging
|
||||
from logging.handlers import RotatingFileHandler
|
||||
|
||||
# Create logs directory
|
||||
os.makedirs("logs", exist_ok=True)
|
||||
|
||||
# Configure logging
|
||||
handler = RotatingFileHandler(
|
||||
"logs/ingestion.log",
|
||||
maxBytes=10_000_000, # 10 MB
|
||||
backupCount=5
|
||||
)
|
||||
handler.setFormatter(logging.Formatter(
|
||||
'%(asctime)s [%(levelname)s] %(name)s: %(message)s'
|
||||
))
|
||||
logger = logging.getLogger()
|
||||
logger.addHandler(handler)
|
||||
```
|
||||
|
||||
### Health Check Endpoint (Optional)
|
||||
|
||||
```python
|
||||
# Add to pote/api/main.py (when building API)
|
||||
from fastapi import FastAPI
|
||||
|
||||
app = FastAPI()
|
||||
|
||||
@app.get("/health")
|
||||
def health_check():
|
||||
from pote.db import SessionLocal
|
||||
from sqlalchemy import text
|
||||
|
||||
try:
|
||||
with SessionLocal() as session:
|
||||
session.execute(text("SELECT 1"))
|
||||
return {"status": "ok", "database": "connected"}
|
||||
except Exception as e:
|
||||
return {"status": "error", "message": str(e)}
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cost Estimates (Monthly)
|
||||
|
||||
| Option | Cost | Notes |
|
||||
|--------|------|-------|
|
||||
| **Local Dev** | $0 | SQLite, your machine |
|
||||
| **VPS (DigitalOcean, Linode)** | $5-12 | Small droplet + managed Postgres |
|
||||
| **AWS (small)** | $20-50 | t3.micro EC2 + db.t3.micro RDS |
|
||||
| **Fly.io / Railway** | $5-15 | Hobby tier, managed |
|
||||
| **Docker on VPS** | $10-20 | One droplet, Docker Compose |
|
||||
|
||||
**Free tier options**:
|
||||
- Railway: Free tier available (limited hours)
|
||||
- Fly.io: Free tier available (limited resources)
|
||||
- Oracle Cloud: Always-free tier (ARM instances)
|
||||
|
||||
---
|
||||
|
||||
## Next Steps After Deployment
|
||||
|
||||
1. **Verify ingestion**: Check logs after first cron run
|
||||
2. **Test queries**: Ensure data is accessible
|
||||
3. **Monitor growth**: Database size, query performance
|
||||
4. **Plan backups**: Set up automated DB dumps
|
||||
5. **Document access**: How to query, who has access
|
||||
|
||||
For Phase 2 (Analytics), you'll add:
|
||||
- Scheduled jobs for computing returns
|
||||
- Clustering jobs (weekly/monthly)
|
||||
- Optional dashboard deployment
|
||||
|
||||
---
|
||||
|
||||
## Quick Deploy (Railway Example)
|
||||
|
||||
Railway is probably the easiest for personal projects:
|
||||
|
||||
```bash
|
||||
# Install Railway CLI
|
||||
npm install -g @railway/cli
|
||||
|
||||
# Login
|
||||
railway login
|
||||
|
||||
# Initialize
|
||||
railway init
|
||||
|
||||
# Add PostgreSQL
|
||||
railway add --database postgres
|
||||
|
||||
# Deploy
|
||||
railway up
|
||||
|
||||
# Add environment variables via dashboard
|
||||
# DATABASE_URL is auto-configured
|
||||
```
|
||||
|
||||
**Cost**: ~$5/mo, scales automatically
|
||||
|
||||
---
|
||||
|
||||
See `docs/05_dev_setup.md` for local development details.
|
||||
|
||||
@@ -0,0 +1,604 @@
|
||||
# Proxmox Deployment Guide
|
||||
|
||||
## Why Proxmox is Perfect for POTE
|
||||
|
||||
✅ **Full control** - Your hardware, your rules
|
||||
✅ **No monthly costs** - Just electricity
|
||||
✅ **Isolated VMs/LXC** - Clean environments
|
||||
✅ **Snapshots** - Easy rollback if needed
|
||||
✅ **Resource efficient** - Run alongside other services
|
||||
|
||||
---
|
||||
|
||||
## Deployment Options on Proxmox
|
||||
|
||||
### Option 1: LXC Container (Recommended) ⭐
|
||||
|
||||
**Pros**: Lightweight, fast, efficient resource usage
|
||||
**Cons**: Linux only (fine for POTE)
|
||||
|
||||
### Option 2: VM with Docker
|
||||
|
||||
**Pros**: Full isolation, can run any OS
|
||||
**Cons**: More resource overhead
|
||||
|
||||
### Option 3: VM without Docker
|
||||
|
||||
**Pros**: Traditional setup, maximum control
|
||||
**Cons**: Manual dependency management
|
||||
|
||||
---
|
||||
|
||||
## Quick Start: LXC Container (Easiest)
|
||||
|
||||
### 1. Create LXC Container
|
||||
|
||||
```bash
|
||||
# In Proxmox web UI or via CLI:
|
||||
|
||||
# Create Ubuntu 22.04 LXC container
|
||||
pct create 100 local:vztmpl/ubuntu-22.04-standard_22.04-1_amd64.tar.zst \
|
||||
--hostname pote \
|
||||
--memory 2048 \
|
||||
--cores 2 \
|
||||
--rootfs local-lvm:8 \
|
||||
--net0 name=eth0,bridge=vmbr0,ip=dhcp \
|
||||
--unprivileged 1 \
|
||||
--features nesting=1
|
||||
|
||||
# Start container
|
||||
pct start 100
|
||||
|
||||
# Enter container
|
||||
pct enter 100
|
||||
```
|
||||
|
||||
Or via Web UI:
|
||||
1. Create CT → Ubuntu 22.04
|
||||
2. Hostname: `pote`
|
||||
3. Memory: 2GB
|
||||
4. Cores: 2
|
||||
5. Disk: 8GB
|
||||
6. Network: Bridge, DHCP
|
||||
|
||||
### 2. Install Dependencies
|
||||
|
||||
```bash
|
||||
# Inside the container
|
||||
apt update && apt upgrade -y
|
||||
|
||||
# Install Python 3.11, PostgreSQL, Git
|
||||
apt install -y python3.11 python3.11-venv python3-pip \
|
||||
postgresql postgresql-contrib git curl
|
||||
|
||||
# Install build tools (for some Python packages)
|
||||
apt install -y build-essential libpq-dev
|
||||
```
|
||||
|
||||
### 3. Setup PostgreSQL
|
||||
|
||||
```bash
|
||||
# Switch to postgres user
|
||||
sudo -u postgres psql
|
||||
|
||||
# Create database and user
|
||||
CREATE DATABASE pote;
|
||||
CREATE USER poteuser WITH PASSWORD 'your_secure_password';
|
||||
GRANT ALL PRIVILEGES ON DATABASE pote TO poteuser;
|
||||
ALTER DATABASE pote OWNER TO poteuser;
|
||||
\q
|
||||
```
|
||||
|
||||
### 4. Clone and Install POTE
|
||||
|
||||
```bash
|
||||
# Create app user (optional but recommended)
|
||||
useradd -m -s /bin/bash poteapp
|
||||
su - poteapp
|
||||
|
||||
# Clone repo
|
||||
git clone https://github.com/your-username/pote.git
|
||||
cd pote
|
||||
|
||||
# Create virtual environment
|
||||
python3.11 -m venv venv
|
||||
source venv/bin/activate
|
||||
|
||||
# Install dependencies
|
||||
pip install --upgrade pip
|
||||
pip install -e .
|
||||
```
|
||||
|
||||
### 5. Configure Environment
|
||||
|
||||
```bash
|
||||
# Create .env file
|
||||
cat > .env << EOF
|
||||
DATABASE_URL=postgresql://poteuser:your_secure_password@localhost:5432/pote
|
||||
QUIVERQUANT_API_KEY=
|
||||
FMP_API_KEY=
|
||||
LOG_LEVEL=INFO
|
||||
EOF
|
||||
|
||||
chmod 600 .env
|
||||
```
|
||||
|
||||
### 6. Run Migrations
|
||||
|
||||
```bash
|
||||
source venv/bin/activate
|
||||
alembic upgrade head
|
||||
```
|
||||
|
||||
### 7. Test Ingestion
|
||||
|
||||
```bash
|
||||
# Test with fixtures (offline)
|
||||
python scripts/ingest_from_fixtures.py
|
||||
|
||||
# Enrich securities
|
||||
python scripts/enrich_securities.py
|
||||
|
||||
# Test with real data (if internet available)
|
||||
python scripts/fetch_congressional_trades.py --days 7
|
||||
```
|
||||
|
||||
### 8. Setup Cron Jobs
|
||||
|
||||
```bash
|
||||
# Edit crontab
|
||||
crontab -e
|
||||
|
||||
# Add these lines:
|
||||
# Fetch trades daily at 6 AM
|
||||
0 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/fetch_congressional_trades.py --days 7 >> /home/poteapp/logs/trades.log 2>&1
|
||||
|
||||
# Enrich securities daily at 6:15 AM
|
||||
15 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/enrich_securities.py >> /home/poteapp/logs/enrich.log 2>&1
|
||||
|
||||
# Update prices daily at 6:30 AM (when built)
|
||||
30 6 * * * cd /home/poteapp/pote && /home/poteapp/pote/venv/bin/python scripts/update_all_prices.py >> /home/poteapp/logs/prices.log 2>&1
|
||||
```
|
||||
|
||||
### 9. Setup Logging
|
||||
|
||||
```bash
|
||||
# Create logs directory
|
||||
mkdir -p /home/poteapp/logs
|
||||
|
||||
# Rotate logs (optional)
|
||||
cat > /etc/logrotate.d/pote << EOF
|
||||
/home/poteapp/logs/*.log {
|
||||
daily
|
||||
rotate 7
|
||||
compress
|
||||
delaycompress
|
||||
missingok
|
||||
notifempty
|
||||
}
|
||||
EOF
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Option 2: VM with Docker (More Isolated)
|
||||
|
||||
### 1. Create VM
|
||||
|
||||
Via Proxmox Web UI:
|
||||
1. Create VM
|
||||
2. OS: Ubuntu Server 22.04
|
||||
3. Memory: 4GB
|
||||
4. Cores: 2
|
||||
5. Disk: 20GB
|
||||
6. Network: Bridge
|
||||
|
||||
### 2. Install Docker
|
||||
|
||||
```bash
|
||||
# SSH into VM
|
||||
ssh user@vm-ip
|
||||
|
||||
# Install Docker
|
||||
curl -fsSL https://get.docker.com -o get-docker.sh
|
||||
sh get-docker.sh
|
||||
|
||||
# Add user to docker group
|
||||
sudo usermod -aG docker $USER
|
||||
newgrp docker
|
||||
|
||||
# Install Docker Compose
|
||||
sudo curl -L "https://github.com/docker/compose/releases/latest/download/docker-compose-$(uname -s)-$(uname -m)" -o /usr/local/bin/docker-compose
|
||||
sudo chmod +x /usr/local/bin/docker-compose
|
||||
```
|
||||
|
||||
### 3. Clone and Deploy
|
||||
|
||||
```bash
|
||||
git clone https://github.com/your-username/pote.git
|
||||
cd pote
|
||||
|
||||
# Create .env
|
||||
cat > .env << EOF
|
||||
POSTGRES_PASSWORD=your_secure_password
|
||||
DATABASE_URL=postgresql://poteuser:your_secure_password@db:5432/pote
|
||||
QUIVERQUANT_API_KEY=
|
||||
FMP_API_KEY=
|
||||
EOF
|
||||
|
||||
# Start services
|
||||
docker-compose up -d
|
||||
|
||||
# Check logs
|
||||
docker-compose logs -f
|
||||
|
||||
# Run migrations
|
||||
docker-compose exec pote alembic upgrade head
|
||||
|
||||
# Test ingestion
|
||||
docker-compose exec pote python scripts/ingest_from_fixtures.py
|
||||
```
|
||||
|
||||
### 4. Setup Auto-start
|
||||
|
||||
```bash
|
||||
# Enable Docker service
|
||||
sudo systemctl enable docker
|
||||
|
||||
# Docker Compose auto-start
|
||||
sudo curl -L https://raw.githubusercontent.com/docker/compose/master/contrib/systemd/docker-compose.service -o /etc/systemd/system/docker-compose@.service
|
||||
|
||||
# Enable for your project
|
||||
sudo systemctl enable docker-compose@pote
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Proxmox-Specific Tips
|
||||
|
||||
### 1. Backups
|
||||
|
||||
```bash
|
||||
# In Proxmox host, backup the container/VM
|
||||
vzdump 100 --mode snapshot --storage local
|
||||
|
||||
# Or via Web UI: Datacenter → Backup → Add
|
||||
# Schedule: Daily, Keep: 7 days
|
||||
```
|
||||
|
||||
### 2. Snapshots
|
||||
|
||||
```bash
|
||||
# Before major changes, take snapshot
|
||||
pct snapshot 100 before-upgrade
|
||||
|
||||
# Rollback if needed
|
||||
pct rollback 100 before-upgrade
|
||||
|
||||
# Or via Web UI: Container → Snapshots
|
||||
```
|
||||
|
||||
### 3. Resource Monitoring
|
||||
|
||||
```bash
|
||||
# Monitor container resources
|
||||
pct status 100
|
||||
pct exec 100 -- df -h
|
||||
pct exec 100 -- free -h
|
||||
|
||||
# Check PostgreSQL size
|
||||
pct exec 100 -- sudo -u postgres psql -c "SELECT pg_size_pretty(pg_database_size('pote'));"
|
||||
```
|
||||
|
||||
### 4. Networking
|
||||
|
||||
**Static IP (Recommended for services)**:
|
||||
```bash
|
||||
# Edit container config on Proxmox host
|
||||
nano /etc/pve/lxc/100.conf
|
||||
|
||||
# Change network config
|
||||
net0: name=eth0,bridge=vmbr0,ip=192.168.1.50/24,gw=192.168.1.1
|
||||
|
||||
# Restart container
|
||||
pct restart 100
|
||||
```
|
||||
|
||||
**Port Forwarding** (if needed for API):
|
||||
```bash
|
||||
# On Proxmox host, forward port 8000 → container
|
||||
iptables -t nat -A PREROUTING -p tcp --dport 8000 -j DNAT --to 192.168.1.50:8000
|
||||
iptables -t nat -A POSTROUTING -j MASQUERADE
|
||||
|
||||
# Make persistent
|
||||
apt install iptables-persistent
|
||||
netfilter-persistent save
|
||||
```
|
||||
|
||||
### 5. Security
|
||||
|
||||
```bash
|
||||
# Inside container, setup firewall
|
||||
apt install ufw
|
||||
|
||||
# Allow SSH
|
||||
ufw allow 22/tcp
|
||||
|
||||
# Allow PostgreSQL (if remote access needed)
|
||||
ufw allow from 192.168.1.0/24 to any port 5432
|
||||
|
||||
# Enable firewall
|
||||
ufw enable
|
||||
```
|
||||
|
||||
### 6. Performance Tuning
|
||||
|
||||
**PostgreSQL** (for LXC with 2GB RAM):
|
||||
```bash
|
||||
# Edit postgresql.conf
|
||||
sudo nano /etc/postgresql/14/main/postgresql.conf
|
||||
|
||||
# Optimize for 2GB RAM
|
||||
shared_buffers = 512MB
|
||||
effective_cache_size = 1536MB
|
||||
maintenance_work_mem = 128MB
|
||||
checkpoint_completion_target = 0.9
|
||||
wal_buffers = 16MB
|
||||
default_statistics_target = 100
|
||||
random_page_cost = 1.1
|
||||
effective_io_concurrency = 200
|
||||
work_mem = 2621kB
|
||||
min_wal_size = 1GB
|
||||
max_wal_size = 4GB
|
||||
|
||||
# Restart PostgreSQL
|
||||
sudo systemctl restart postgresql
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Resource Requirements
|
||||
|
||||
### Minimum (Development/Testing)
|
||||
- **Memory**: 1GB
|
||||
- **Cores**: 1
|
||||
- **Disk**: 5GB
|
||||
- **Network**: Bridged
|
||||
|
||||
### Recommended (Production)
|
||||
- **Memory**: 2-4GB
|
||||
- **Cores**: 2
|
||||
- **Disk**: 20GB (with room for logs/backups)
|
||||
- **Network**: Bridged with static IP
|
||||
|
||||
### With Dashboard (Phase 3)
|
||||
- **Memory**: 4GB
|
||||
- **Cores**: 2-4
|
||||
- **Disk**: 20GB
|
||||
|
||||
---
|
||||
|
||||
## Monitoring & Maintenance
|
||||
|
||||
### 1. Check Service Health
|
||||
|
||||
```bash
|
||||
# Database connection
|
||||
pct exec 100 -- sudo -u poteapp bash -c 'cd /home/poteapp/pote && source venv/bin/activate && python -c "from pote.db import SessionLocal; from sqlalchemy import text; s = SessionLocal(); s.execute(text(\"SELECT 1\")); print(\"DB OK\")"'
|
||||
|
||||
# Check last ingestion
|
||||
pct exec 100 -- sudo -u postgres psql pote -c "SELECT COUNT(*), MAX(created_at) FROM trades;"
|
||||
|
||||
# Check disk usage
|
||||
pct exec 100 -- df -h
|
||||
|
||||
# Check logs
|
||||
pct exec 100 -- tail -f /home/poteapp/logs/trades.log
|
||||
```
|
||||
|
||||
### 2. Database Maintenance
|
||||
|
||||
```bash
|
||||
# Backup database
|
||||
pct exec 100 -- sudo -u postgres pg_dump pote > pote_backup_$(date +%Y%m%d).sql
|
||||
|
||||
# Vacuum (clean up)
|
||||
pct exec 100 -- sudo -u postgres psql pote -c "VACUUM ANALYZE;"
|
||||
|
||||
# Check database size
|
||||
pct exec 100 -- sudo -u postgres psql -c "SELECT pg_size_pretty(pg_database_size('pote'));"
|
||||
```
|
||||
|
||||
### 3. Update POTE
|
||||
|
||||
```bash
|
||||
# Enter container
|
||||
pct enter 100
|
||||
su - poteapp
|
||||
cd pote
|
||||
|
||||
# Pull latest code
|
||||
git pull
|
||||
|
||||
# Update dependencies
|
||||
source venv/bin/activate
|
||||
pip install --upgrade -e .
|
||||
|
||||
# Run migrations
|
||||
alembic upgrade head
|
||||
|
||||
# Test
|
||||
python scripts/ingest_from_fixtures.py
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Troubleshooting
|
||||
|
||||
### Container won't start
|
||||
```bash
|
||||
# Check logs
|
||||
pct status 100
|
||||
journalctl -u pve-container@100
|
||||
|
||||
# Try start with debug
|
||||
pct start 100 --debug
|
||||
```
|
||||
|
||||
### PostgreSQL connection issues
|
||||
```bash
|
||||
# Check if PostgreSQL is running
|
||||
pct exec 100 -- systemctl status postgresql
|
||||
|
||||
# Check connections
|
||||
pct exec 100 -- sudo -u postgres psql -c "SELECT * FROM pg_stat_activity;"
|
||||
|
||||
# Reset password if needed
|
||||
pct exec 100 -- sudo -u postgres psql -c "ALTER USER poteuser PASSWORD 'new_password';"
|
||||
```
|
||||
|
||||
### Out of disk space
|
||||
```bash
|
||||
# Check usage
|
||||
pct exec 100 -- df -h
|
||||
|
||||
# Clean logs
|
||||
pct exec 100 -- find /home/poteapp/logs -name "*.log" -mtime +7 -delete
|
||||
|
||||
# Clean apt cache
|
||||
pct exec 100 -- apt clean
|
||||
|
||||
# Resize container disk (on Proxmox host)
|
||||
lvresize -L +5G /dev/pve/vm-100-disk-0
|
||||
pct resize 100 rootfs +5G
|
||||
```
|
||||
|
||||
### Python package issues
|
||||
```bash
|
||||
# Reinstall in venv
|
||||
pct exec 100 -- sudo -u poteapp bash -c 'cd /home/poteapp/pote && rm -rf venv && python3.11 -m venv venv && source venv/bin/activate && pip install -e .'
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
## Cost Analysis
|
||||
|
||||
### Proxmox LXC (Your Setup)
|
||||
- **Hardware**: Already owned
|
||||
- **Power**: ~$5-15/mo (depends on your setup)
|
||||
- **Internet**: Existing connection
|
||||
- **Total**: **~$10/mo** (just power)
|
||||
|
||||
vs.
|
||||
|
||||
- **VPS**: $10-20/mo
|
||||
- **Cloud**: $20-50/mo
|
||||
- **Managed**: $50-100/mo
|
||||
|
||||
**Your Proxmox = 50-90% cost savings!**
|
||||
|
||||
---
|
||||
|
||||
## Next Steps
|
||||
|
||||
1. ✅ Create LXC container
|
||||
2. ✅ Install dependencies
|
||||
3. ✅ Setup PostgreSQL
|
||||
4. ✅ Deploy POTE
|
||||
5. ✅ Configure cron jobs
|
||||
6. ✅ Setup backups
|
||||
7. ⏭️ Build Phase 2 (Analytics)
|
||||
8. ⏭️ Add FastAPI dashboard (optional)
|
||||
|
||||
---
|
||||
|
||||
## Example: Complete Setup Script
|
||||
|
||||
Save this as `proxmox_setup.sh` in your container:
|
||||
|
||||
```bash
|
||||
#!/bin/bash
|
||||
set -e
|
||||
|
||||
echo "=== POTE Proxmox Setup ==="
|
||||
|
||||
# Update system
|
||||
echo "Updating system..."
|
||||
apt update && apt upgrade -y
|
||||
|
||||
# Install dependencies
|
||||
echo "Installing dependencies..."
|
||||
apt install -y python3.11 python3.11-venv python3-pip \
|
||||
postgresql postgresql-contrib git curl \
|
||||
build-essential libpq-dev
|
||||
|
||||
# Setup PostgreSQL
|
||||
echo "Setting up PostgreSQL..."
|
||||
sudo -u postgres psql << EOF
|
||||
CREATE DATABASE pote;
|
||||
CREATE USER poteuser WITH PASSWORD 'changeme123';
|
||||
GRANT ALL PRIVILEGES ON DATABASE pote TO poteuser;
|
||||
ALTER DATABASE pote OWNER TO poteuser;
|
||||
EOF
|
||||
|
||||
# Create app user
|
||||
echo "Creating app user..."
|
||||
useradd -m -s /bin/bash poteapp || true
|
||||
|
||||
# Clone repo
|
||||
echo "Cloning POTE..."
|
||||
sudo -u poteapp git clone https://github.com/your-username/pote.git /home/poteapp/pote || true
|
||||
|
||||
# Setup Python environment
|
||||
echo "Setting up Python environment..."
|
||||
sudo -u poteapp bash << 'EOF'
|
||||
cd /home/poteapp/pote
|
||||
python3.11 -m venv venv
|
||||
source venv/bin/activate
|
||||
pip install --upgrade pip
|
||||
pip install -e .
|
||||
EOF
|
||||
|
||||
# Create .env
|
||||
echo "Creating .env..."
|
||||
sudo -u poteapp bash << 'EOF'
|
||||
cat > /home/poteapp/pote/.env << ENVEOF
|
||||
DATABASE_URL=postgresql://poteuser:changeme123@localhost:5432/pote
|
||||
QUIVERQUANT_API_KEY=
|
||||
FMP_API_KEY=
|
||||
LOG_LEVEL=INFO
|
||||
ENVEOF
|
||||
chmod 600 /home/poteapp/pote/.env
|
||||
EOF
|
||||
|
||||
# Run migrations
|
||||
echo "Running migrations..."
|
||||
sudo -u poteapp bash << 'EOF'
|
||||
cd /home/poteapp/pote
|
||||
source venv/bin/activate
|
||||
alembic upgrade head
|
||||
EOF
|
||||
|
||||
# Create logs directory
|
||||
sudo -u poteapp mkdir -p /home/poteapp/logs
|
||||
|
||||
echo ""
|
||||
echo "✅ Setup complete!"
|
||||
echo ""
|
||||
echo "Next steps:"
|
||||
echo "1. su - poteapp"
|
||||
echo "2. cd pote && source venv/bin/activate"
|
||||
echo "3. python scripts/ingest_from_fixtures.py"
|
||||
echo "4. Setup cron jobs (see docs/08_proxmox_deployment.md)"
|
||||
```
|
||||
|
||||
Run it:
|
||||
```bash
|
||||
chmod +x proxmox_setup.sh
|
||||
./proxmox_setup.sh
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Your Proxmox setup gives you enterprise-grade infrastructure at hobby costs!** 🚀
|
||||
|
||||
@@ -0,0 +1,80 @@
|
||||
# PR1 Summary: Project Scaffold + DB + Price Loader
|
||||
|
||||
**Status**: ✅ Complete
|
||||
**Date**: 2025-12-13
|
||||
|
||||
## What was built
|
||||
|
||||
### 1. Project scaffold
|
||||
- `pyproject.toml` with all dependencies (SQLAlchemy, Alembic, yfinance, pandas, pytest, ruff, black, etc.)
|
||||
- `src/pote/` layout with config, db, and ingestion modules
|
||||
- `.gitignore`, `.env.example`, `Makefile` for dev workflow
|
||||
- Docs: `README.md` + 6 `.md` files in `docs/` covering MVP, architecture, schema, sources, safety/ethics, and dev setup
|
||||
|
||||
### 2. Database models (SQLAlchemy 2.0)
|
||||
- **Officials**: Congress members (name, chamber, party, state, bioguide_id)
|
||||
- **Securities**: stocks/bonds (ticker, name, exchange, sector)
|
||||
- **Trades**: disclosed transactions (official_id, security_id, transaction_date, filing_date, side, value ranges)
|
||||
- **Prices**: daily OHLCV (security_id, date, open/high/low/close/volume)
|
||||
- **Metrics stubs**: `metrics_official` and `metrics_trade` (Phase 2)
|
||||
|
||||
Includes proper indexes, unique constraints, and relationships.
|
||||
|
||||
### 3. Alembic migrations
|
||||
- Initialized Alembic with `env.py` wired to our config
|
||||
- Generated and applied initial migration (`66fd166195e8`)
|
||||
- DB file: `pote.db` (SQLite for dev)
|
||||
|
||||
### 4. Price loader (`PriceLoader`)
|
||||
- Fetches daily price data from **yfinance**
|
||||
- Idempotent: skips existing dates, resumes from gaps
|
||||
- Upsert logic (insert or update on conflict)
|
||||
- Handles single ticker or bulk fetches
|
||||
- Logging + basic error handling
|
||||
|
||||
### 5. Tests (pytest)
|
||||
- `tests/conftest.py`: fixtures for in-memory DB, sample officials/securities/trades/prices
|
||||
- `tests/test_models.py`: model creation, relationships, unique constraints, queries (7 tests)
|
||||
- `tests/test_price_loader.py`: loader logic, idempotency, upsert, mocking yfinance (8 tests)
|
||||
- **Result**: 15 tests, all passing ✅
|
||||
|
||||
### 6. Tooling
|
||||
- **Black** + **ruff** configured and run (all code formatted + linted)
|
||||
- `Makefile` with targets: `install`, `test`, `lint`, `format`, `migrate`, `clean`
|
||||
- Smoke-test script: `scripts/fetch_sample_prices.py` (verified live with AAPL/MSFT/TSLA)
|
||||
|
||||
## What works now
|
||||
- You can spin up the DB, run migrations, fetch price data, and query it
|
||||
- All core Phase 1 foundations are in place
|
||||
- Tests confirm models and ingestion work correctly
|
||||
|
||||
## Next steps (PR2+)
|
||||
Per `docs/00_mvp.md`:
|
||||
- **PR2**: QuiverQuant or FMP client for Congress trades
|
||||
- **PR3**: ETL job to populate `officials` and `trades` tables
|
||||
- **PR4+**: Analytics (abnormal returns, clustering, signals)
|
||||
|
||||
## How to run
|
||||
```bash
|
||||
# Install
|
||||
make install
|
||||
source venv/bin/activate
|
||||
|
||||
# Run migrations
|
||||
make migrate
|
||||
|
||||
# Fetch sample prices
|
||||
python scripts/fetch_sample_prices.py
|
||||
|
||||
# Run tests
|
||||
make test
|
||||
|
||||
# Lint + format
|
||||
make lint
|
||||
make format
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Research-only reminder**: This tool is for transparency and descriptive analytics using public data. Not investment advice.
|
||||
|
||||
@@ -0,0 +1,161 @@
|
||||
# PR2 Summary: Congressional Trade Ingestion
|
||||
|
||||
**Status**: ✅ Complete
|
||||
**Date**: 2025-12-14
|
||||
|
||||
## What was built
|
||||
|
||||
### 1. House Stock Watcher Client (`src/pote/ingestion/house_watcher.py`)
|
||||
- Free API client for https://housestockwatcher.com
|
||||
- No authentication required
|
||||
- Methods:
|
||||
- `fetch_all_transactions(limit)`: Get all recent transactions
|
||||
- `fetch_recent_transactions(days)`: Filter to last N days
|
||||
- Helper functions:
|
||||
- `parse_amount_range()`: Parse "$1,001 - $15,000" → (min, max)
|
||||
- `normalize_transaction_type()`: "Purchase" → "buy", "Sale" → "sell"
|
||||
|
||||
### 2. Trade Loader ETL (`src/pote/ingestion/trade_loader.py`)
|
||||
- `TradeLoader.ingest_transactions()`: Full ETL pipeline
|
||||
- Get-or-create logic for officials and securities (deduplication)
|
||||
- Upsert trades by source + external_id (no duplicates)
|
||||
- Returns counts: `{"officials": N, "securities": N, "trades": N}`
|
||||
- Proper error handling and logging
|
||||
|
||||
### 3. Test Fixtures
|
||||
- `tests/fixtures/sample_house_watcher.json`: 5 realistic sample transactions
|
||||
- Includes House + Senate, Democrats + Republicans, various tickers
|
||||
|
||||
### 4. Tests (13 new tests, all passing ✅)
|
||||
**`tests/test_house_watcher.py` (8 tests)**:
|
||||
- Amount range parsing (with range, single value, invalid)
|
||||
- Transaction type normalization
|
||||
- Fetching all/recent transactions (mocked)
|
||||
- Client context manager
|
||||
|
||||
**`tests/test_trade_loader.py` (5 tests)**:
|
||||
- Ingest from fixture file (full integration)
|
||||
- Duplicate transaction handling (idempotency)
|
||||
- Missing ticker handling (skip gracefully)
|
||||
- Senate vs House official creation
|
||||
- Multiple trades for same official
|
||||
|
||||
### 5. Smoke-test Script (`scripts/fetch_congressional_trades.py`)
|
||||
- CLI tool to fetch live data from House Stock Watcher
|
||||
- Options: `--days N`, `--limit N`, `--all`
|
||||
- Ingests into DB and shows summary stats
|
||||
- Usage:
|
||||
```bash
|
||||
python scripts/fetch_congressional_trades.py --days 30
|
||||
python scripts/fetch_congressional_trades.py --all --limit 100
|
||||
```
|
||||
|
||||
## What works now
|
||||
|
||||
### Live Data Ingestion (FREE!)
|
||||
```bash
|
||||
# Fetch last 30 days of congressional trades
|
||||
python scripts/fetch_congressional_trades.py --days 30
|
||||
|
||||
# Sample output:
|
||||
# ✓ Officials created/updated: 47
|
||||
# ✓ Securities created/updated: 89
|
||||
# ✓ Trades ingested: 234
|
||||
```
|
||||
|
||||
### Database Queries
|
||||
```python
|
||||
from pote.db import SessionLocal
|
||||
from pote.db.models import Official, Trade
|
||||
from sqlalchemy import select
|
||||
|
||||
with SessionLocal() as session:
|
||||
# Find Nancy Pelosi's trades
|
||||
stmt = select(Official).where(Official.name == "Nancy Pelosi")
|
||||
pelosi = session.scalars(stmt).first()
|
||||
|
||||
stmt = select(Trade).where(Trade.official_id == pelosi.id)
|
||||
trades = session.scalars(stmt).all()
|
||||
print(f"Pelosi has {len(trades)} trades")
|
||||
```
|
||||
|
||||
### Test Coverage
|
||||
```bash
|
||||
make test
|
||||
# 28 tests passed in 1.23s
|
||||
# Coverage: 87%+
|
||||
```
|
||||
|
||||
## Data Model Updates
|
||||
|
||||
No schema changes! Existing tables work perfectly:
|
||||
- `officials`: Populated from House Stock Watcher API
|
||||
- `securities`: Tickers from trades (name=ticker for now, will enrich later)
|
||||
- `trades`: Full trade records with transaction_date, filing_date, side, value ranges
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
1. **Free API First**: House Stock Watcher = $0, no rate limits
|
||||
2. **Idempotency**: Re-running ingestion won't create duplicates
|
||||
3. **Graceful Degradation**: Skip trades with missing tickers, log warnings
|
||||
4. **Tuple Returns**: `_get_or_create_*` methods return `(entity, is_new)` for accurate counting
|
||||
5. **External IDs**: `official_id_security_id_date_side` for deduplication
|
||||
|
||||
## Performance
|
||||
|
||||
- Fetches 100+ transactions in ~2 seconds
|
||||
- Ingest 100 transactions in ~0.5 seconds (SQLite)
|
||||
- Tests run in 1.2 seconds (28 tests)
|
||||
|
||||
## Next Steps (PR3+)
|
||||
|
||||
Per `docs/00_mvp.md`:
|
||||
- **PR3**: Enrich securities with yfinance (fetch names, sectors, exchanges)
|
||||
- **PR4**: Abnormal return calculations
|
||||
- **PR5**: Clustering & signals
|
||||
- **PR6**: Optional FastAPI + dashboard
|
||||
|
||||
## How to Use
|
||||
|
||||
### 1. Fetch Live Data
|
||||
```bash
|
||||
# Recent trades (last 7 days)
|
||||
python scripts/fetch_congressional_trades.py --days 7
|
||||
|
||||
# All trades, limited to 50
|
||||
python scripts/fetch_congressional_trades.py --all --limit 50
|
||||
```
|
||||
|
||||
### 2. Programmatic Usage
|
||||
```python
|
||||
from pote.db import SessionLocal
|
||||
from pote.ingestion.house_watcher import HouseWatcherClient
|
||||
from pote.ingestion.trade_loader import TradeLoader
|
||||
|
||||
with HouseWatcherClient() as client:
|
||||
txns = client.fetch_recent_transactions(days=30)
|
||||
|
||||
with SessionLocal() as session:
|
||||
loader = TradeLoader(session)
|
||||
counts = loader.ingest_transactions(txns)
|
||||
print(f"Ingested {counts['trades']} trades")
|
||||
```
|
||||
|
||||
### 3. Run Tests
|
||||
```bash
|
||||
# All tests
|
||||
make test
|
||||
|
||||
# Just trade ingestion tests
|
||||
pytest tests/test_trade_loader.py -v
|
||||
|
||||
# With coverage
|
||||
pytest tests/ --cov=pote --cov-report=term-missing
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Cost**: $0 (uses free House Stock Watcher API)
|
||||
**Dependencies**: `httpx` (already in `pyproject.toml`)
|
||||
**Research-only reminder**: This tool is for transparency and descriptive analytics. Not investment advice.
|
||||
|
||||
@@ -0,0 +1,226 @@
|
||||
# PR3 Summary: Security Enrichment + Deployment
|
||||
|
||||
**Status**: ✅ Complete
|
||||
**Date**: 2025-12-14
|
||||
|
||||
## What was built
|
||||
|
||||
### 1. Security Enrichment (`src/pote/ingestion/security_enricher.py`)
|
||||
- `SecurityEnricher` class for enriching securities with yfinance data
|
||||
- Fetches: company names, sectors, industries, exchanges
|
||||
- Detects asset type: stock, ETF, mutual fund, index
|
||||
- Methods:
|
||||
- `enrich_security(security, force)`: Enrich single security
|
||||
- `enrich_all_securities(limit, force)`: Batch enrichment
|
||||
- `enrich_by_ticker(ticker)`: Enrich specific ticker
|
||||
- Smart skipping: only enriches unenriched securities (unless `force=True`)
|
||||
|
||||
### 2. Enrichment Script (`scripts/enrich_securities.py`)
|
||||
- CLI tool for enriching securities
|
||||
- Usage:
|
||||
```bash
|
||||
# Enrich all unenriched securities
|
||||
python scripts/enrich_securities.py
|
||||
|
||||
# Enrich specific ticker
|
||||
python scripts/enrich_securities.py --ticker AAPL
|
||||
|
||||
# Limit batch size
|
||||
python scripts/enrich_securities.py --limit 10
|
||||
|
||||
# Force re-enrichment
|
||||
python scripts/enrich_securities.py --force
|
||||
```
|
||||
|
||||
### 3. Tests (9 new tests, all passing ✅)
|
||||
**`tests/test_security_enricher.py`**:
|
||||
- Successful enrichment with complete data
|
||||
- ETF detection and classification
|
||||
- Skip already enriched securities
|
||||
- Force refresh functionality
|
||||
- Handle missing/invalid data gracefully
|
||||
- Batch enrichment
|
||||
- Enrichment with limit
|
||||
- Enrich by specific ticker
|
||||
- Handle ticker not found
|
||||
|
||||
### 4. Deployment Infrastructure
|
||||
- **`Dockerfile`**: Production-ready container image
|
||||
- **`docker-compose.yml`**: Full stack (app + PostgreSQL)
|
||||
- **`.dockerignore`**: Optimize image size
|
||||
- **`docs/07_deployment.md`**: Comprehensive deployment guide
|
||||
- Local development (SQLite)
|
||||
- Single server (PostgreSQL + cron)
|
||||
- Docker deployment
|
||||
- Cloud deployment (AWS, Fly.io, Railway)
|
||||
- Cost estimates
|
||||
- Production checklist
|
||||
|
||||
## What works now
|
||||
|
||||
### Enrich Securities from Fixtures
|
||||
```bash
|
||||
# Our existing fixtures have these tickers: NVDA, MSFT, AAPL, TSLA, GOOGL
|
||||
# They're created as "unenriched" (name == ticker)
|
||||
|
||||
python scripts/enrich_securities.py
|
||||
|
||||
# Output:
|
||||
# Enriching 5 securities
|
||||
# Enriched NVDA: NVIDIA Corporation (Technology)
|
||||
# Enriched MSFT: Microsoft Corporation (Technology)
|
||||
# Enriched AAPL: Apple Inc. (Technology)
|
||||
# Enriched TSLA: Tesla, Inc. (Consumer Cyclical)
|
||||
# Enriched GOOGL: Alphabet Inc. (Communication Services)
|
||||
# ✓ Successfully enriched: 5
|
||||
```
|
||||
|
||||
### Query Enriched Data
|
||||
```python
|
||||
from pote.db import SessionLocal
|
||||
from pote.db.models import Security
|
||||
from sqlalchemy import select
|
||||
|
||||
with SessionLocal() as session:
|
||||
stmt = select(Security).where(Security.sector.isnot(None))
|
||||
enriched = session.scalars(stmt).all()
|
||||
|
||||
for sec in enriched:
|
||||
print(f"{sec.ticker}: {sec.name} ({sec.sector})")
|
||||
```
|
||||
|
||||
### Docker Deployment
|
||||
```bash
|
||||
# Quick start
|
||||
docker-compose up -d
|
||||
|
||||
# Run migrations
|
||||
docker-compose exec pote alembic upgrade head
|
||||
|
||||
# Ingest trades from fixtures (offline)
|
||||
docker-compose exec pote python scripts/ingest_from_fixtures.py
|
||||
|
||||
# Enrich securities (needs network in container)
|
||||
docker-compose exec pote python scripts/enrich_securities.py
|
||||
```
|
||||
|
||||
## Data Model Updates
|
||||
|
||||
No schema changes! The `securities` table already had all necessary fields:
|
||||
- `name`: Now populated with full company name
|
||||
- `sector`: Technology, Healthcare, Finance, etc.
|
||||
- `industry`: Specific industry within sector
|
||||
- `exchange`: NASDAQ, NYSE, etc.
|
||||
- `asset_type`: stock, etf, mutual_fund, index
|
||||
|
||||
## Key Design Decisions
|
||||
|
||||
1. **Smart Skipping**: Only enrich securities where `name == ticker` (unenriched)
|
||||
2. **Force Option**: Can re-enrich with `--force` flag
|
||||
3. **Graceful Degradation**: Skip/log if yfinance data unavailable
|
||||
4. **Batch Control**: `--limit` for rate limiting or testing
|
||||
5. **Asset Type Detection**: Automatically classify ETFs, mutual funds, indexes
|
||||
|
||||
## Performance
|
||||
|
||||
- Enrich single security: ~1 second (yfinance API call)
|
||||
- Batch enrichment: ~1-2 seconds per security
|
||||
- Recommendation: Run weekly or when new tickers appear
|
||||
- yfinance is free but rate-limited (be reasonable!)
|
||||
|
||||
## Integration with Existing System
|
||||
|
||||
### After Trade Ingestion
|
||||
```python
|
||||
# In production cron job:
|
||||
# 1. Fetch trades
|
||||
python scripts/fetch_congressional_trades.py --days 7
|
||||
|
||||
# 2. Enrich any new securities
|
||||
python scripts/enrich_securities.py
|
||||
|
||||
# 3. Fetch prices for all securities
|
||||
python scripts/update_all_prices.py # To be built in PR4
|
||||
```
|
||||
|
||||
### Cron Schedule (Production)
|
||||
```bash
|
||||
# Daily at 6 AM: Fetch trades
|
||||
0 6 * * * cd /path/to/pote && venv/bin/python scripts/fetch_congressional_trades.py --days 7
|
||||
|
||||
# Daily at 6:15 AM: Enrich new securities
|
||||
15 6 * * * cd /path/to/pote && venv/bin/python scripts/enrich_securities.py
|
||||
|
||||
# Daily at 6:30 AM: Update prices
|
||||
30 6 * * * cd /path/to/pote && venv/bin/python scripts/update_all_prices.py
|
||||
```
|
||||
|
||||
## Deployment Options
|
||||
|
||||
| Option | Complexity | Cost/month | Best For |
|
||||
|--------|-----------|------------|----------|
|
||||
| **Local** | ⭐ | $0 | Development |
|
||||
| **VPS + Docker** | ⭐⭐ | $10-20 | Personal deployment |
|
||||
| **Railway/Fly.io** | ⭐ | $5-15 | Easy cloud |
|
||||
| **AWS** | ⭐⭐⭐ | $20-50 | Scalable production |
|
||||
|
||||
See [`docs/07_deployment.md`](07_deployment.md) for detailed guides.
|
||||
|
||||
## Next Steps (PR4+)
|
||||
|
||||
Per `docs/00_mvp.md`:
|
||||
- **PR4**: Analytics - abnormal returns, benchmarks
|
||||
- **PR5**: Clustering & signals
|
||||
- **PR6**: FastAPI + dashboard
|
||||
|
||||
## How to Use
|
||||
|
||||
### 1. Enrich All Securities
|
||||
```bash
|
||||
python scripts/enrich_securities.py
|
||||
```
|
||||
|
||||
### 2. Enrich Specific Ticker
|
||||
```bash
|
||||
python scripts/enrich_securities.py --ticker NVDA
|
||||
```
|
||||
|
||||
### 3. Re-enrich Everything
|
||||
```bash
|
||||
python scripts/enrich_securities.py --force
|
||||
```
|
||||
|
||||
### 4. Programmatic Usage
|
||||
```python
|
||||
from pote.db import SessionLocal
|
||||
from pote.ingestion.security_enricher import SecurityEnricher
|
||||
|
||||
with SessionLocal() as session:
|
||||
enricher = SecurityEnricher(session)
|
||||
|
||||
# Enrich all unenriched
|
||||
counts = enricher.enrich_all_securities()
|
||||
print(f"Enriched {counts['enriched']} securities")
|
||||
|
||||
# Enrich specific ticker
|
||||
enricher.enrich_by_ticker("AAPL")
|
||||
```
|
||||
|
||||
## Test Coverage
|
||||
|
||||
```bash
|
||||
pytest tests/ -v
|
||||
|
||||
# 37 tests passing
|
||||
# Coverage: 87%+
|
||||
|
||||
# New tests:
|
||||
# - test_security_enricher.py (9 tests)
|
||||
```
|
||||
|
||||
---
|
||||
|
||||
**Cost**: Still $0 (yfinance is free!)
|
||||
**Dependencies**: yfinance (already in `pyproject.toml`)
|
||||
**Research-only reminder**: This tool is for transparency and descriptive analytics. Not investment advice.
|
||||
|
||||
Reference in New Issue
Block a user