outreach/PROJECT.md

🏗️ Outreach Engine - Technical Architecture

📋 Project Overview

The Outreach Engine is a Node.js-based email automation system designed for professional outreach campaigns to law firms. It features a modular architecture with comprehensive testing, tracking, and safety mechanisms.

🏛️ System Architecture

Core Components

┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Main App      │    │   Template      │    │   Database      │
│   (index.js)    │◄──►│   Engine        │◄──►│   (SQLite)      │
└─────────────────┘    └─────────────────┘    └─────────────────┘
         │                       │                       │
         ▼                       ▼                       ▼
┌─────────────────┐    ┌─────────────────┐    ┌─────────────────┐
│   Rate Limiter  │    │   Attachment    │    │   Tracking      │
│   & Error       │    │   Handler       │    │   Server        │
│   Handler       │    └─────────────────┘    └─────────────────┘
└─────────────────┘

Data Flow

1. Initialization: Load config → Initialize DB → Load firm data
2. Email Processing: Template rendering → Rate limiting → SMTP sending
3. Tracking: Open/click events → Database logging → Analytics
4. Error Handling: Retry logic → Fallback mechanisms → Logging

📁 Folder Structure

outreach-engine/
├── 📄 index.js                    # Main application entry point
├── 📁 config/
│   └── 📄 index.js                # Environment configuration management
├── 📁 lib/                        # Core libraries
│   ├── 📄 attachmentHandler.js    # File attachment processing
│   ├── 📄 database.js             # SQLite database operations
│   ├── 📄 errorHandler.js         # Error handling and retry logic
│   ├── 📄 logger.js               # Winston logging configuration
│   ├── 📄 rateLimiter.js          # Email rate limiting
│   ├── 📄 templateEngine.js       # Handlebars template processing
│   └── 📄 trackingServer.js       # HTTP server for tracking
├── 📁 templates/                  # Email templates (Handlebars)
│   ├── 📄 outreach.html           # Main outreach template
│   ├── 📄 campaign-1-saas.html    # SaaS campaign
│   ├── 📄 campaign-2-data-service.html
│   ├── 📄 campaign-3-license.html
│   ├── 📄 campaign-4-gui.html
│   ├── 📄 general-intro.html
│   └── 📄 employment-layer.html
├── 📁 tests/                      # Test suite
│   ├── 📁 lib/
│   │   ├── 📄 errorHandler.test.js
│   │   ├── 📄 rateLimiter.test.js
│   │   └── 📄 templateEngine.test.js
│   ├── 📄 setup.js                # Test configuration
│   ├── 📄 jest.config.js          # Jest configuration
│   └── 📄 test-campaigns.json     # Test campaign data
├── 📁 scripts/                    # Utility scripts
│   ├── 📄 migrate-to-database.js  # JSON to database migration
│   └── 📄 run-campaigns.js        # Campaign testing script
├── 📁 logs/                       # Application logs
├── 📁 data/                       # Generated data files
├── 📁 coverage/                   # Test coverage reports
├── 📄 firm.json                   # Law firm contact data
├── 📄 tests/test-campaigns.json   # Test campaign data
├── 📄 package.json                # Dependencies and scripts
└── 📄 tests/jest.config.js        # Jest testing configuration

🔧 Dependencies

Production Dependencies

Package	Version	Purpose
nodemailer	^7.0.5	Email sending via SMTP
handlebars	^4.7.8	Template engine for emails
sqlite	^5.1.1	SQLite database driver
sqlite3	^5.1.7	SQLite3 bindings
winston	^3.17.0	Structured logging
dotenv	^17.2.0	Environment variable loading
delay	^6.0.0	Promise-based delays

Development Dependencies

Package	Version	Purpose
jest	^30.0.4	Testing framework
@jest/globals	^30.0.4	Jest global functions

🗄️ Database Schema

Tables

firms

CREATE TABLE firms (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  firm_name TEXT NOT NULL,
  location TEXT,
  website TEXT,
  contact_email TEXT UNIQUE,
  state TEXT,
  created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

email_sends

CREATE TABLE email_sends (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  campaign_id TEXT,
  firm_id INTEGER,
  recipient_email TEXT,
  subject TEXT,
  status TEXT,
  tracking_id TEXT,
  sent_at DATETIME DEFAULT CURRENT_TIMESTAMP,
  FOREIGN KEY (firm_id) REFERENCES firms (id)
);

tracking_events

CREATE TABLE tracking_events (
  id INTEGER PRIMARY KEY AUTOINCREMENT,
  tracking_id TEXT,
  event_type TEXT, -- 'open' or 'click'
  ip_address TEXT,
  user_agent TEXT,
  url TEXT,
  created_at DATETIME DEFAULT CURRENT_TIMESTAMP
);

🔄 Core Workflows

1. Application Startup

graph TD
    A[Start] --> B[Load Environment]
    B --> C[Initialize Database]
    C --> D[Load Firm Data]
    D --> E[Start Tracking Server]
    E --> F[Begin Email Campaign]

2. Email Sending Process

graph TD
    A[Get Next Firm] --> B[Rate Limit Check]
    B --> C[Render Template]
    C --> D[Add Tracking]
    D --> E[Send via SMTP]
    E --> F[Log Success/Failure]
    F --> G[Wait Delay]
    G --> A

3. Template Processing

graph TD
    A[Load Template] --> B[Compile Handlebars]
    B --> C[Inject Firm Data]
    C --> D[Add GIF if enabled]
    D --> E[Generate Text Version]
    E --> F[Add Tracking Pixels]
    F --> G[Return Email Content]

🧪 Testing Architecture

Test Structure

• Unit Tests: tests/lib/*.test.js - Test individual modules
• Integration Tests: Built into main application
• Coverage: Jest generates coverage reports in coverage/

Test Categories

1. Error Handler Tests: Retry logic, error classification
2. Rate Limiter Tests: Delay calculations, pause logic
3. Template Engine Tests: Template rendering, data injection

Test Commands

npm test              # Run all tests
npm run test:watch    # Watch mode
npm run test:coverage # Generate coverage report

⚙️ Configuration System

Environment Loading

• Auto-detection: Based on NODE_ENV
• Files: .env.development, .env.production
• Fallback: Default values for missing variables

Configuration Categories

1. Email Settings: SMTP, credentials, test mode
2. Rate Limiting: Delays, limits, pause intervals
3. Tracking: Server port, domain, enable/disable
4. Logging: Levels, debug mode, file rotation
5. Templates: GIF settings, attachment handling

🔒 Security & Safety

Test Mode Protection

• Fail-safe validation: Stops if test recipient not set
• Double verification: Checks test mode twice before sending
• Clear logging: Every email shows test status
• No production data: Test mode never uses real firm emails

Rate Limiting

• Configurable delays: 5+ minutes between emails
• Hourly limits: 15 emails/hour default
• Automatic pauses: Every 50 emails, pause 30 minutes
• Randomization: Adds jitter to prevent detection

📊 Monitoring & Analytics

Logging System

• Winston: Structured logging with multiple transports
• File rotation: Automatic log file management
• Debug mode: Detailed SMTP and template information
• Error tracking: Comprehensive error logging

Tracking Analytics

• Open tracking: Invisible pixel tracking
• Click tracking: URL wrapping with redirects
• Database storage: All events stored in SQLite
• Real-time server: HTTP server on port 3000

🚀 Performance Considerations

Optimization Features

• Database indexing: Fast queries on email and tracking
• Template caching: Handlebars templates compiled once
• Batch processing: Configurable batch sizes
• Memory management: Proper cleanup of resources

Scalability

• Modular design: Easy to extend with new features
• Database abstraction: Can switch to other databases
• Template system: Easy to add new email templates
• Configuration: Environment-based settings

🔧 Development Workflow

Code Organization

• Separation of concerns: Each module has single responsibility
• Error boundaries: Comprehensive error handling
• Async/await: Modern JavaScript patterns
• ES6 modules: Clean import/export structure

Testing Strategy

• Unit tests: Test individual functions
• Integration tests: Test module interactions
• End-to-end: Test complete email sending flow
• Coverage goals: Maintain high test coverage

📈 Future Enhancements

Planned Features

• A/B testing: Template variant testing
• Advanced analytics: Dashboard for campaign metrics
• API endpoints: REST API for external integration
• Web interface: Admin dashboard for campaign management

Technical Debt

• TypeScript migration: Add type safety
• Dependency updates: Keep packages current
• Performance optimization: Database query optimization
• Security hardening: Additional security measures

🐛 Debugging Guide

Common Issues

1. SMTP errors: Check credentials and 2FA settings
2. Template errors: Verify Handlebars syntax
3. Database errors: Check SQLite file permissions
4. Rate limiting: Adjust delay settings

Debug Tools

• Debug mode: DEBUG=true npm start
• Log files: Check logs/ directory
• Database inspection: Use SQLite browser
• Network monitoring: Check tracking server logs

📚 API Reference

Main Functions

• sendEmails(): Main email sending loop
• sendSingleEmail(): Individual email sending
• initializeData(): Database and data setup
• processFirm(): Single firm processing

Library Functions

• templateEngine.render(): Template rendering
• rateLimiter.check(): Rate limit validation
• errorHandler.handle(): Error processing
• database.logEmailSend(): Email logging

---

This document provides comprehensive technical information for development, debugging, and AI assistance. For user-focused information, see README.md.