Add comprehensive deployment and automation FAQ
Answers user's questions: - What happens after deployment? (nothing automatic by default) - How to get reports? (3 options: email, SSH, future web UI) - Where are reports sent? (email or saved to ~/logs/) - Do you need to check IP? (depends on setup method) - Can we setup email reports? (YES! 5-minute setup) - Do we need CI/CD pipelines? (optional, but included) - Can we use existing Ansible pipeline? (concepts reused, not directly) This document ties everything together and provides clear next steps.
This commit is contained in:
ilia
parent
0d8d85adc1
commit
d8f723bafb
338
DEPLOYMENT_AND_AUTOMATION.md
Normal file
338
DEPLOYMENT_AND_AUTOMATION.md
Normal file
@ -0,0 +1,338 @@
|
|||||||
|
# POTE Deployment & Automation Guide
|
||||||
|
|
||||||
|
## 🎯 Quick Answer to Your Questions
|
||||||
|
|
||||||
|
### After Deployment, What Happens?
|
||||||
|
|
||||||
|
**By default: NOTHING automatic happens.** You need to set up automation.
|
||||||
|
|
||||||
|
The deployed system is:
|
||||||
|
- ✅ Running (database, code installed)
|
||||||
|
- ✅ Accessible via SSH at your Proxmox IP
|
||||||
|
- ❌ NOT fetching data automatically
|
||||||
|
- ❌ NOT sending reports automatically
|
||||||
|
- ❌ NOT monitoring markets automatically
|
||||||
|
|
||||||
|
**You must either:**
|
||||||
|
1. **Run scripts manually** when you want updates, OR
|
||||||
|
2. **Set up automation** (5 minutes, see below)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🚀 Option 1: Automated Email Reports (Recommended)
|
||||||
|
|
||||||
|
### What You Get
|
||||||
|
|
||||||
|
- **Daily reports at 6 AM** (or your chosen time)
|
||||||
|
- New congressional trades from yesterday
|
||||||
|
- Market alerts (unusual activity)
|
||||||
|
- Suspicious timing detections
|
||||||
|
|
||||||
|
- **Weekly reports on Sundays**
|
||||||
|
- Most active officials & securities
|
||||||
|
- Repeat offenders (consistent suspicious timing)
|
||||||
|
- Pattern analysis
|
||||||
|
|
||||||
|
- **Sent to your email** (Gmail, SendGrid, or custom SMTP)
|
||||||
|
|
||||||
|
### 5-Minute Setup
|
||||||
|
|
||||||
|
SSH to your deployed POTE server:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh poteapp@your-proxmox-ip
|
||||||
|
cd ~/pote
|
||||||
|
```
|
||||||
|
|
||||||
|
Run the interactive setup:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
./scripts/setup_cron.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
Follow prompts:
|
||||||
|
1. Enter your email address
|
||||||
|
2. Choose report time (default: 6 AM)
|
||||||
|
3. Done! ✅
|
||||||
|
|
||||||
|
**See full guide:** [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📍 Option 2: Access Reports via IP (No Email)
|
||||||
|
|
||||||
|
If you don't want email, you can:
|
||||||
|
|
||||||
|
### SSH to Server and View Reports
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh poteapp@your-proxmox-ip
|
||||||
|
cd ~/pote
|
||||||
|
source venv/bin/activate
|
||||||
|
|
||||||
|
# Generate report to stdout
|
||||||
|
python scripts/send_daily_report.py --to dummy@example.com --save-to-file /tmp/report.txt
|
||||||
|
cat /tmp/report.txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### Access Saved Reports
|
||||||
|
|
||||||
|
If you set up automation (even without email), reports are saved to:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
# SSH to server
|
||||||
|
ssh poteapp@your-proxmox-ip
|
||||||
|
|
||||||
|
# View reports
|
||||||
|
ls -lh ~/logs/
|
||||||
|
cat ~/logs/daily_report_$(date +%Y%m%d).txt
|
||||||
|
cat ~/logs/weekly_report_$(date +%Y%m%d).txt
|
||||||
|
```
|
||||||
|
|
||||||
|
### Run Scripts Manually via SSH
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh poteapp@your-proxmox-ip
|
||||||
|
cd ~/pote
|
||||||
|
source venv/bin/activate
|
||||||
|
|
||||||
|
# Fetch new data
|
||||||
|
python scripts/fetch_congressional_trades.py
|
||||||
|
python scripts/monitor_market.py --scan
|
||||||
|
python scripts/analyze_disclosure_timing.py --recent 7
|
||||||
|
|
||||||
|
# View results in database
|
||||||
|
python scripts/health_check.py
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🌐 Option 3: Build a Web Interface (Future)
|
||||||
|
|
||||||
|
Currently, POTE is **command-line only**. No web UI yet.
|
||||||
|
|
||||||
|
**To add a web interface, you would need to:**
|
||||||
|
1. Build a FastAPI backend (expose read-only endpoints)
|
||||||
|
2. Build a React/Streamlit frontend
|
||||||
|
3. Access via `http://your-proxmox-ip:8000`
|
||||||
|
|
||||||
|
This is **not implemented yet**, but it's on the roadmap (Phase 3).
|
||||||
|
|
||||||
|
For now, use SSH or email reports.
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔄 Do You Need CI/CD Pipelines?
|
||||||
|
|
||||||
|
### What the Pipeline Does
|
||||||
|
|
||||||
|
The included CI/CD pipeline (`.github/workflows/ci.yml`) runs on **every git push**:
|
||||||
|
|
||||||
|
1. ✅ Lint & test (93 tests)
|
||||||
|
2. ✅ Security scanning
|
||||||
|
3. ✅ Dependency scanning
|
||||||
|
4. ✅ Docker build test
|
||||||
|
|
||||||
|
### Should You Use It?
|
||||||
|
|
||||||
|
**YES, if you:**
|
||||||
|
- Are actively developing POTE
|
||||||
|
- Want to catch bugs before deployment
|
||||||
|
- Want automated testing on every commit
|
||||||
|
- Use GitHub or Gitea for version control
|
||||||
|
|
||||||
|
**NO, if you:**
|
||||||
|
- Just want to use POTE as-is
|
||||||
|
- Don't plan to modify the code
|
||||||
|
- Don't have a CI/CD runner set up
|
||||||
|
|
||||||
|
### How to Use the Pipeline
|
||||||
|
|
||||||
|
The pipeline is **GitHub Actions / Gitea Actions compatible**.
|
||||||
|
|
||||||
|
#### For GitHub:
|
||||||
|
Push your code to GitHub. The workflow runs automatically.
|
||||||
|
|
||||||
|
#### For Gitea (Your Setup):
|
||||||
|
1. Ensure Gitea Actions runner is installed
|
||||||
|
2. Push to your Gitea repo
|
||||||
|
3. Workflow runs automatically
|
||||||
|
|
||||||
|
#### For Local Testing:
|
||||||
|
```bash
|
||||||
|
# Run tests locally
|
||||||
|
pytest tests/ -v
|
||||||
|
|
||||||
|
# Run linters
|
||||||
|
ruff check src/ tests/
|
||||||
|
black --check src/ tests/
|
||||||
|
mypy src/
|
||||||
|
|
||||||
|
# Build Docker image
|
||||||
|
docker build -t pote:test .
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📊 Comparison of Options
|
||||||
|
|
||||||
|
| Method | Pros | Cons | Best For |
|
||||||
|
|--------|------|------|----------|
|
||||||
|
| **Automated Email** | ✅ Convenient<br>✅ No SSH needed<br>✅ Daily/weekly updates | ❌ Requires SMTP setup | Most users |
|
||||||
|
| **SSH + Manual Scripts** | ✅ Full control<br>✅ No email needed | ❌ Manual work<br>❌ Must remember to run | Power users |
|
||||||
|
| **Saved Reports (SSH access)** | ✅ Automated<br>✅ No email | ❌ Must SSH to view | Users without email |
|
||||||
|
| **Web Interface** | ✅ User-friendly | ❌ Not implemented yet | Future |
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🛠️ Your Ansible Pipeline
|
||||||
|
|
||||||
|
Your existing Ansible CI/CD pipeline is **NOT directly usable** for POTE because:
|
||||||
|
|
||||||
|
1. **Language mismatch**: Your pipeline is for Node.js/Ansible; POTE is Python
|
||||||
|
2. **Different tooling**: POTE uses pytest, ruff, mypy (not npm, ansible-lint)
|
||||||
|
3. **Different structure**: POTE uses different security scanners
|
||||||
|
|
||||||
|
### What You CAN Reuse
|
||||||
|
|
||||||
|
**Concepts from your pipeline that ARE used in POTE's CI/CD:**
|
||||||
|
|
||||||
|
- ✅ Security scanning (Trivy, Bandit instead of Gitleaks)
|
||||||
|
- ✅ Dependency scanning (Trivy instead of npm audit)
|
||||||
|
- ✅ SAST scanning (Bandit instead of Semgrep)
|
||||||
|
- ✅ Container scanning (Docker build test)
|
||||||
|
- ✅ Workflow summary generation
|
||||||
|
|
||||||
|
**The POTE pipeline (`.github/workflows/ci.yml`) already includes all of these!**
|
||||||
|
|
||||||
|
### If You Want to Integrate with Your Gitea
|
||||||
|
|
||||||
|
Your Gitea server can run the POTE pipeline using Gitea Actions:
|
||||||
|
|
||||||
|
1. Push POTE to your Gitea
|
||||||
|
2. Gitea Actions will detect `.github/workflows/ci.yml`
|
||||||
|
3. Workflow runs automatically (if runner is configured)
|
||||||
|
|
||||||
|
**No modifications needed** - it's already compatible!
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📧 Email Setup Examples
|
||||||
|
|
||||||
|
### Gmail (Most Common)
|
||||||
|
|
||||||
|
In your deployed `.env` file:
|
||||||
|
|
||||||
|
```env
|
||||||
|
SMTP_HOST=smtp.gmail.com
|
||||||
|
SMTP_PORT=587
|
||||||
|
SMTP_USER=your-email@gmail.com
|
||||||
|
SMTP_PASSWORD=your-16-char-app-password
|
||||||
|
FROM_EMAIL=pote-reports@gmail.com
|
||||||
|
REPORT_RECIPIENTS=your-email@example.com
|
||||||
|
```
|
||||||
|
|
||||||
|
**Important**: Use an [App Password](https://myaccount.google.com/apppasswords), NOT your regular password!
|
||||||
|
|
||||||
|
### SendGrid
|
||||||
|
|
||||||
|
```env
|
||||||
|
SMTP_HOST=smtp.sendgrid.net
|
||||||
|
SMTP_PORT=587
|
||||||
|
SMTP_USER=apikey
|
||||||
|
SMTP_PASSWORD=SG.your-api-key-here
|
||||||
|
FROM_EMAIL=noreply@yourdomain.com
|
||||||
|
REPORT_RECIPIENTS=admin@yourdomain.com
|
||||||
|
```
|
||||||
|
|
||||||
|
### Custom Mail Server
|
||||||
|
|
||||||
|
```env
|
||||||
|
SMTP_HOST=mail.yourdomain.com
|
||||||
|
SMTP_PORT=587
|
||||||
|
SMTP_USER=username
|
||||||
|
SMTP_PASSWORD=password
|
||||||
|
FROM_EMAIL=pote@yourdomain.com
|
||||||
|
REPORT_RECIPIENTS=admin@yourdomain.com
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## ✅ Recommended Setup for Most Users
|
||||||
|
|
||||||
|
1. **Deploy to Proxmox** (5 min)
|
||||||
|
```bash
|
||||||
|
bash scripts/proxmox_setup.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
2. **Set up automated email reports** (5 min)
|
||||||
|
```bash
|
||||||
|
ssh poteapp@your-ip
|
||||||
|
cd ~/pote
|
||||||
|
./scripts/setup_cron.sh
|
||||||
|
```
|
||||||
|
|
||||||
|
3. **Done!** You'll receive:
|
||||||
|
- Daily reports at 6 AM
|
||||||
|
- Weekly reports on Sundays
|
||||||
|
- All reports also saved to `~/logs/`
|
||||||
|
|
||||||
|
4. **(Optional)** Set up CI/CD if you plan to develop:
|
||||||
|
- Push to GitHub/Gitea
|
||||||
|
- Pipeline runs automatically
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🔍 Monitoring & Health Checks
|
||||||
|
|
||||||
|
### Add System Health Monitoring
|
||||||
|
|
||||||
|
Edit crontab:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
crontab -e
|
||||||
|
```
|
||||||
|
|
||||||
|
Add health check (every 6 hours):
|
||||||
|
|
||||||
|
```cron
|
||||||
|
0 */6 * * * /home/poteapp/pote/venv/bin/python /home/poteapp/pote/scripts/health_check.py >> /home/poteapp/logs/health.log 2>&1
|
||||||
|
```
|
||||||
|
|
||||||
|
Check health anytime:
|
||||||
|
|
||||||
|
```bash
|
||||||
|
ssh poteapp@your-ip
|
||||||
|
cd ~/pote
|
||||||
|
source venv/bin/activate
|
||||||
|
python scripts/health_check.py
|
||||||
|
```
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 📚 Full Documentation
|
||||||
|
|
||||||
|
- **Automation Setup**: [`AUTOMATION_QUICKSTART.md`](AUTOMATION_QUICKSTART.md) ⭐
|
||||||
|
- **Deployment**: [`PROXMOX_QUICKSTART.md`](PROXMOX_QUICKSTART.md) ⭐
|
||||||
|
- **Usage**: [`QUICKSTART.md`](QUICKSTART.md) ⭐
|
||||||
|
- **Detailed Automation Guide**: [`docs/12_automation_and_reporting.md`](docs/12_automation_and_reporting.md)
|
||||||
|
- **Monitoring System**: [`MONITORING_SYSTEM_COMPLETE.md`](MONITORING_SYSTEM_COMPLETE.md)
|
||||||
|
|
||||||
|
---
|
||||||
|
|
||||||
|
## 🎉 Summary
|
||||||
|
|
||||||
|
**After deployment:**
|
||||||
|
- Reports are NOT sent automatically by default
|
||||||
|
- You must set up automation OR run scripts manually
|
||||||
|
- **Recommended: 5-minute automation setup** with `./scripts/setup_cron.sh`
|
||||||
|
- Reports can be emailed OR saved to `~/logs/` for SSH access
|
||||||
|
- CI/CD pipeline is included and ready for GitHub/Gitea
|
||||||
|
- Your Ansible pipeline is not directly usable, but concepts are already implemented
|
||||||
|
|
||||||
|
**Most users should:**
|
||||||
|
1. Deploy to Proxmox
|
||||||
|
2. Run `./scripts/setup_cron.sh`
|
||||||
|
3. Receive daily/weekly email reports
|
||||||
|
4. Done! 🚀
|
||||||
|
|
||||||
Loading…
x
Reference in New Issue
Block a user