From d8f723bafbfa01a8f39dfb4d987d76fbc99a7fe4 Mon Sep 17 00:00:00 2001 From: ilia Date: Mon, 15 Dec 2025 15:35:33 -0500 Subject: [PATCH] Add comprehensive deployment and automation FAQ Answers user's questions: - What happens after deployment? (nothing automatic by default) - How to get reports? (3 options: email, SSH, future web UI) - Where are reports sent? (email or saved to ~/logs/) - Do you need to check IP? (depends on setup method) - Can we setup email reports? (YES! 5-minute setup) - Do we need CI/CD pipelines? (optional, but included) - Can we use existing Ansible pipeline? (concepts reused, not directly) This document ties everything together and provides clear next steps. --- DEPLOYMENT_AND_AUTOMATION.md | 338 +++++++++++++++++++++++++++++++++++ 1 file changed, 338 insertions(+) create mode 100644 DEPLOYMENT_AND_AUTOMATION.md diff --git a/DEPLOYMENT_AND_AUTOMATION.md b/DEPLOYMENT_AND_AUTOMATION.md new file mode 100644 index 0000000..e1961e7 --- /dev/null +++ b/DEPLOYMENT_AND_AUTOMATION.md @@ -0,0 +1,338 @@ +# POTE Deployment & Automation Guide + +## 🎯 Quick Answer to Your Questions + +### After Deployment, What Happens? + +**By default: NOTHING automatic happens.** You need to set up automation. + +The deployed system is: +- ✅ Running (database, code installed) +- ✅ Accessible via SSH at your Proxmox IP +- ❌ NOT fetching data automatically +- ❌ NOT sending reports automatically +- ❌ NOT monitoring markets automatically + +**You must either:** +1. **Run scripts manually** when you want updates, OR +2. **Set up automation** (5 minutes, see below) + +--- + +## 🚀 Option 1: Automated Email Reports (Recommended) + +### What You Get + +- **Daily reports at 6 AM** (or your chosen time) + - New congressional trades from yesterday + - Market alerts (unusual activity) + - Suspicious timing detections + +- **Weekly reports on Sundays** + - Most active officials & securities + - Repeat offenders (consistent suspicious timing) + - Pattern analysis + +- **Sent to your email** (Gmail, SendGrid, or custom SMTP) + +### 5-Minute Setup + +SSH to your deployed POTE server: + +```bash +ssh poteapp@your-proxmox-ip +cd ~/pote +``` + +Run the interactive setup: + +```bash +./scripts/setup_cron.sh +``` + +Follow prompts: +1. Enter your email address +2. Choose report time (default: 6 AM) +3. Done! ✅ + +**See full guide:** [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md) + +--- + +## 📍 Option 2: Access Reports via IP (No Email) + +If you don't want email, you can: + +### SSH to Server and View Reports + +```bash +ssh poteapp@your-proxmox-ip +cd ~/pote +source venv/bin/activate + +# Generate report to stdout +python scripts/send_daily_report.py --to dummy@example.com --save-to-file /tmp/report.txt +cat /tmp/report.txt +``` + +### Access Saved Reports + +If you set up automation (even without email), reports are saved to: + +```bash +# SSH to server +ssh poteapp@your-proxmox-ip + +# View reports +ls -lh ~/logs/ +cat ~/logs/daily_report_$(date +%Y%m%d).txt +cat ~/logs/weekly_report_$(date +%Y%m%d).txt +``` + +### Run Scripts Manually via SSH + +```bash +ssh poteapp@your-proxmox-ip +cd ~/pote +source venv/bin/activate + +# Fetch new data +python scripts/fetch_congressional_trades.py +python scripts/monitor_market.py --scan +python scripts/analyze_disclosure_timing.py --recent 7 + +# View results in database +python scripts/health_check.py +``` + +--- + +## 🌐 Option 3: Build a Web Interface (Future) + +Currently, POTE is **command-line only**. No web UI yet. + +**To add a web interface, you would need to:** +1. Build a FastAPI backend (expose read-only endpoints) +2. Build a React/Streamlit frontend +3. Access via `http://your-proxmox-ip:8000` + +This is **not implemented yet**, but it's on the roadmap (Phase 3). + +For now, use SSH or email reports. + +--- + +## 🔄 Do You Need CI/CD Pipelines? + +### What the Pipeline Does + +The included CI/CD pipeline (`.github/workflows/ci.yml`) runs on **every git push**: + +1. ✅ Lint & test (93 tests) +2. ✅ Security scanning +3. ✅ Dependency scanning +4. ✅ Docker build test + +### Should You Use It? + +**YES, if you:** +- Are actively developing POTE +- Want to catch bugs before deployment +- Want automated testing on every commit +- Use GitHub or Gitea for version control + +**NO, if you:** +- Just want to use POTE as-is +- Don't plan to modify the code +- Don't have a CI/CD runner set up + +### How to Use the Pipeline + +The pipeline is **GitHub Actions / Gitea Actions compatible**. + +#### For GitHub: +Push your code to GitHub. The workflow runs automatically. + +#### For Gitea (Your Setup): +1. Ensure Gitea Actions runner is installed +2. Push to your Gitea repo +3. Workflow runs automatically + +#### For Local Testing: +```bash +# Run tests locally +pytest tests/ -v + +# Run linters +ruff check src/ tests/ +black --check src/ tests/ +mypy src/ + +# Build Docker image +docker build -t pote:test . +``` + +--- + +## 📊 Comparison of Options + +| Method | Pros | Cons | Best For | +|--------|------|------|----------| +| **Automated Email** | ✅ Convenient
✅ No SSH needed
✅ Daily/weekly updates | ❌ Requires SMTP setup | Most users | +| **SSH + Manual Scripts** | ✅ Full control
✅ No email needed | ❌ Manual work
❌ Must remember to run | Power users | +| **Saved Reports (SSH access)** | ✅ Automated
✅ No email | ❌ Must SSH to view | Users without email | +| **Web Interface** | ✅ User-friendly | ❌ Not implemented yet | Future | + +--- + +## 🛠️ Your Ansible Pipeline + +Your existing Ansible CI/CD pipeline is **NOT directly usable** for POTE because: + +1. **Language mismatch**: Your pipeline is for Node.js/Ansible; POTE is Python +2. **Different tooling**: POTE uses pytest, ruff, mypy (not npm, ansible-lint) +3. **Different structure**: POTE uses different security scanners + +### What You CAN Reuse + +**Concepts from your pipeline that ARE used in POTE's CI/CD:** + +- ✅ Security scanning (Trivy, Bandit instead of Gitleaks) +- ✅ Dependency scanning (Trivy instead of npm audit) +- ✅ SAST scanning (Bandit instead of Semgrep) +- ✅ Container scanning (Docker build test) +- ✅ Workflow summary generation + +**The POTE pipeline (`.github/workflows/ci.yml`) already includes all of these!** + +### If You Want to Integrate with Your Gitea + +Your Gitea server can run the POTE pipeline using Gitea Actions: + +1. Push POTE to your Gitea +2. Gitea Actions will detect `.github/workflows/ci.yml` +3. Workflow runs automatically (if runner is configured) + +**No modifications needed** - it's already compatible! + +--- + +## 📧 Email Setup Examples + +### Gmail (Most Common) + +In your deployed `.env` file: + +```env +SMTP_HOST=smtp.gmail.com +SMTP_PORT=587 +SMTP_USER=your-email@gmail.com +SMTP_PASSWORD=your-16-char-app-password +FROM_EMAIL=pote-reports@gmail.com +REPORT_RECIPIENTS=your-email@example.com +``` + +**Important**: Use an [App Password](https://myaccount.google.com/apppasswords), NOT your regular password! + +### SendGrid + +```env +SMTP_HOST=smtp.sendgrid.net +SMTP_PORT=587 +SMTP_USER=apikey +SMTP_PASSWORD=SG.your-api-key-here +FROM_EMAIL=noreply@yourdomain.com +REPORT_RECIPIENTS=admin@yourdomain.com +``` + +### Custom Mail Server + +```env +SMTP_HOST=mail.yourdomain.com +SMTP_PORT=587 +SMTP_USER=username +SMTP_PASSWORD=password +FROM_EMAIL=pote@yourdomain.com +REPORT_RECIPIENTS=admin@yourdomain.com +``` + +--- + +## ✅ Recommended Setup for Most Users + +1. **Deploy to Proxmox** (5 min) + ```bash + bash scripts/proxmox_setup.sh + ``` + +2. **Set up automated email reports** (5 min) + ```bash + ssh poteapp@your-ip + cd ~/pote + ./scripts/setup_cron.sh + ``` + +3. **Done!** You'll receive: + - Daily reports at 6 AM + - Weekly reports on Sundays + - All reports also saved to `~/logs/` + +4. **(Optional)** Set up CI/CD if you plan to develop: + - Push to GitHub/Gitea + - Pipeline runs automatically + +--- + +## 🔍 Monitoring & Health Checks + +### Add System Health Monitoring + +Edit crontab: + +```bash +crontab -e +``` + +Add health check (every 6 hours): + +```cron +0 */6 * * * /home/poteapp/pote/venv/bin/python /home/poteapp/pote/scripts/health_check.py >> /home/poteapp/logs/health.log 2>&1 +``` + +Check health anytime: + +```bash +ssh poteapp@your-ip +cd ~/pote +source venv/bin/activate +python scripts/health_check.py +``` + +--- + +## 📚 Full Documentation + +- **Automation Setup**: [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md) ⭐ +- **Deployment**: [`PROXMOX_QUICKSTART.md`](PROXMOX_QUICKSTART.md) ⭐ +- **Usage**: [`QUICKSTART.md`](QUICKSTART.md) ⭐ +- **Detailed Automation Guide**: [`docs/12_automation_and_reporting.md`](docs/12_automation_and_reporting.md) +- **Monitoring System**: [`MONITORING_SYSTEM_COMPLETE.md`](MONITORING_SYSTEM_COMPLETE.md) + +--- + +## 🎉 Summary + +**After deployment:** +- Reports are NOT sent automatically by default +- You must set up automation OR run scripts manually +- **Recommended: 5-minute automation setup** with `./scripts/setup_cron.sh` +- Reports can be emailed OR saved to `~/logs/` for SSH access +- CI/CD pipeline is included and ready for GitHub/Gitea +- Your Ansible pipeline is not directly usable, but concepts are already implemented + +**Most users should:** +1. Deploy to Proxmox +2. Run `./scripts/setup_cron.sh` +3. Receive daily/weekly email reports +4. Done! 🚀 +