5 Commits

Author	SHA1	Message	Date
ilia	f94ca17b39	Fix CI: Install Node.js in containers for checkout action ... ISSUE: ====== Gitea Actions containers need Node.js to run actions/checkout@v4 Error: 'exec: "node": executable file not found in $PATH' FIX: ==== Added Node.js installation step before checkout in all jobs: For Debian/Ubuntu containers (python:3.11-bullseye, ubuntu:22.04): apt-get update && apt-get install -y curl git curl -fsSL https://deb.nodesource.com/setup_20.x | bash - apt-get install -y nodejs For Alpine containers (gitleaks, trivy): apk add --no-cache nodejs npm curl git JOBS FIXED: =========== ✅ lint-and-test ✅ secret-scanning ✅ security-scan ✅ dependency-scan ✅ sast-scan ✅ container-scan ✅ docker-build-test (no container, already works) This is a common requirement for Gitea Actions when using custom containers.	2025-12-24 22:53:33 -05:00
ilia	d40b412f67	Remove Ansible code and enhance CI pipeline with security scanning ... REMOVED: ======== ❌ ansible/ directory (moved to infrastructure repo) ❌ ANSIBLE_INTEGRATION.md (redundant) KEPT (for reference): ===================== ✅ ANSIBLE_HANDOFF.md - Integration guide for Ansible team ✅ ANSIBLE_TECHNICAL_REFERENCE.md - Exact commands/paths for role ✅ CUSTOMIZATION_CHECKLIST.md - Configuration reference ✅ MOVE_ANSIBLE_TO_SEPARATE_REPO.md - Migration guide ENHANCED CI PIPELINE: ===================== Added comprehensive security scanning: 🔐 Secret Scanning (Gitleaks) - Scans for exposed credentials, API keys, tokens - Checks entire git history - Redacted output for safety 🔒 Security Scan (Safety + Bandit) - Safety: Known vulnerabilities in Python dependencies - Bandit: Static security analysis of Python code - Detects common security issues 📦 Dependency Scan (Trivy) - Scans all dependencies for vulnerabilities - Checks Python packages and system libraries - CVE database lookup 🔍 SAST Scan (Semgrep) - Static Application Security Testing - Language-aware pattern matching - Detects security anti-patterns 🐳 Container Scan (Trivy) - Scans Dockerfile for misconfigurations - Filesystem vulnerability scanning - HIGH/CRITICAL severity focus 🐋 Docker Build Test - Ensures Docker image builds successfully - Tests basic import functionality - Uses build cache for speed 📊 Workflow Summary - Comprehensive status report - Shows all security layers - Easy-to-read summary RATIONALE: ========== Ansible code belongs in infrastructure repo, not app repo. This eliminates circular dependency and follows best practices. Enhanced CI provides multiple layers of security validation.	2025-12-24 22:33:20 -05:00
ilia	3910ca9d04	Update CI to run on all branches (main, qa, dev)	2025-12-24 21:53:14 -05:00
ilia	ead0820cf9	Add Gitea Secrets integration for CI/CD and deployment ... NEW FEATURES: ============ 📁 GITEA_SECRETS_GUIDE.md: - Comprehensive guide on using Gitea secrets - Store passwords in Gitea (not in git!) - Use in CI/CD and deployment workflows - Best practices and security recommendations 🔧 .github/workflows/ci.yml (UPDATED): - Now uses Gitea secrets with fallbacks - ${{ secrets.SMTP_PASSWORD || 'testpass123' }} - ${{ secrets.DB_PASSWORD || 'testpass123' }} - Tests run with real credentials from Gitea 🚀 .github/workflows/deploy.yml (NEW): - Automated deployment to Proxmox - Manual trigger via Gitea UI - Steps: 1. SSH to Proxmox with secrets.PROXMOX_SSH_KEY 2. Pull latest code 3. Update .env with secrets from Gitea 4. Run migrations 5. Health check 6. Test email 7. Rollback on failure HOW IT WORKS: ============= 1. Store passwords in Gitea (Settings → Secrets) 2. CI/CD uses secrets automatically 3. Deployment workflow updates .env on Proxmox 4. Best of both worlds: secure CI + simple runtime SECRETS TO ADD IN GITEA: ======================== - SMTP_PASSWORD: your mail password - DB_PASSWORD: changeme123 - PROXMOX_HOST: 10.0.10.95 - PROXMOX_USER: poteapp - PROXMOX_SSH_KEY: (SSH private key) - SMTP_HOST: mail.levkin.ca - SMTP_USER: test@levkin.ca - FROM_EMAIL: test@levkin.ca USAGE: ====== # In Gitea UI: Actions → Deploy to Proxmox → Run workflow # Or push commits: git push origin main # CI runs with secrets automatically See GITEA_SECRETS_GUIDE.md for full instructions!	2025-12-15 15:52:19 -05:00
ilia	0d8d85adc1	Add complete automation, reporting, and CI/CD system ... Features Added: ============== 📧 EMAIL REPORTING SYSTEM: - EmailReporter: Send reports via SMTP (Gmail, SendGrid, custom) - ReportGenerator: Generate daily/weekly summaries with HTML/text formatting - Configurable via .env (SMTP_HOST, SMTP_PORT, etc.) - Scripts: send_daily_report.py, send_weekly_report.py 🤖 AUTOMATED RUNS: - automated_daily_run.sh: Full daily ETL pipeline + reporting - automated_weekly_run.sh: Weekly pattern analysis + reports - setup_cron.sh: Interactive cron job setup (5-minute setup) - Logs saved to ~/logs/ with automatic cleanup 🔍 HEALTH CHECKS: - health_check.py: System health monitoring - Checks: DB connection, data freshness, counts, recent alerts - JSON output for programmatic use - Exit codes for monitoring integration 🚀 CI/CD PIPELINE: - .github/workflows/ci.yml: Full CI/CD pipeline - GitHub Actions / Gitea Actions compatible - Jobs: lint & test, security scan, dependency scan, Docker build - PostgreSQL service for integration tests - 93 tests passing in CI 📚 COMPREHENSIVE DOCUMENTATION: - AUTOMATION_QUICKSTART.md: 5-minute email setup guide - docs/12_automation_and_reporting.md: Full automation guide - Updated README.md with automation links - Deployment → Production workflow guide 🛠️ IMPROVEMENTS: - All shell scripts made executable - Environment variable examples in .env.example - Report logs saved with timestamps - 30-day log retention with auto-cleanup - Health checks can be scheduled via cron WHAT THIS ENABLES: ================== After deployment, users can: 1. Set up automated daily/weekly email reports (5 min) 2. Receive HTML+text emails with: - New trades, market alerts, suspicious timing - Weekly patterns, rankings, repeat offenders 3. Monitor system health automatically 4. Run full CI/CD pipeline on every commit 5. Deploy with confidence (tests + security scans) USAGE: ====== # One-time setup (on deployed server) ./scripts/setup_cron.sh # Or manually send reports python scripts/send_daily_report.py --to user@example.com python scripts/send_weekly_report.py --to user@example.com # Check system health python scripts/health_check.py See AUTOMATION_QUICKSTART.md for full instructions. 93 tests passing | Full CI/CD | Email reports ready	2025-12-15 15:34:31 -05:00